Gmail - Parser - HTTP - SFPT - trying to download CSV

How To
, ,
1

Hello,

am trying to extract the link from email and download the contect, which is CSV on that URL

I setp this Scenario:

  • Gmail - check for specific email based on Subject
  • Parser - extract URL from HTML, I did that by “Type = Link (a)” and “HTML content” in html field
  • HTTP - now this is where the problem is :frowning:

Because when I click on the link in the Gmail or any other browser I get CSV I can download.

When using HTTP in make (or wget on server) for that extracted URL, that works in regular browser, it does not work here and I get:

  • statusCode

400

  • headersArray

  • cookieHeadersArray

  • Data

<?xml version="1.0" encoding="UTF-8"?> InvalidRequestRequest specific response headers cannot be used for anonymous GET requests.95M2NM9JQTBBMJAJoKnxT5ND6qcvFL0So84J4X36+NniPoV+5NeDLL/UFHZsb1GT3I65i69KhtGs73DWz8SKnCMtm/E=
  • fileSize

308

the file is hosted on Amazon Cloud I guess and provided by Salsidy service.

How can I proceed so I download the file successfully using Make scenarios?

Thanks a lot,
Bohdan

2

Hi @Bohdan_Skovronsky and welcome to the Make Community!

It’s possible that the link you are getting is obfuscated and goes through a few rewrites before you have access to the “real” link. ANd to get to that, you need to maintain session information, which is what a browser does but an API call might not.

Are you at liberty to share one of these URLs so someone can test it tos see what’s happening behind the scenes?

L

3

Sure, it is just an older stocklist, here you can try, it obviously holds some “keys” that are ok to be transferred via browser but not via straightforward link… what I need is to get that CSv file.

the thing is that it is being sent by email on a daily basis, that is why I need to extract it, copy to my FPT and then work with it as with other feeds.

here is the link:
https://salsify-dandelion.s3-external-1.amazonaws.com/1/6884/user_data_export/1/2024/09/18/03%3A35%3A48-37ef4112/export.csv?response-content-disposition=attachment%3B%20filename%3D"Bloomingville%20Daily%20CSV%20Inventory%20Update.csv"&response-content-type=text%2Fcsv&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIARRX37GWVL5YUBHGV%2F20240918%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240918T033548Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=d59051020c49849bd5c25cafb050aa2f492a5842d43b1922d05f5f0a5ecca303

thanks for your help.

4

Hmmm…

I’m not sure what the solution is.

I did manage to make it work with curl like this:

curl "https://salsify-dandelion.s3-external-1.amazonaws.com/1/6884/user_data_export/1/2024/09/18/03%3A35%3A48-37ef4112/export.csv?response-content-disposition=attachment%3B%20filename%3D%22Bloomingville%20Daily%20CSV%20Inventory%20Update.csv%22&response-content-type=text%2Fcsv&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIARRX37GWVL5YUBHGV%2F20240918%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240918T033548Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=d59051020c49849bd5c25cafb050aa2f492a5842d43b1922d05f5f0a5ecca303"

I got as far as this:

InvalidConfigurationError
Error: 403 Forbidden
{
  "Error": {
    "Code": [
      "SignatureDoesNotMatch"
    ],
    "Message": [
      "The request signature we calculated does not match the signature you provided. Check your key and signing method."
    ],
    "AWSAccessKeyId": [
      "AKIARRX37GWVL5YUBHGV"
    ],
    "StringToSign": [
      "AWS4-HMAC-SHA256\n20240918T033548Z\n20240918/us-east-1/s3/aws4_request\n7b38230b4832af42b5b419e756d3e019a9b87473c53100eae32908a232511beb"
    ],
    "SignatureProvided": [
      "d59051020c49849bd5c25cafb050aa2f492a5842d43b1922d05f5f0a5ecca303"
    ],
    "StringToSignBytes": [
      "41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 34 30 39 31 38 54 30 33 33 35 34 38 5a 0a 32 30 32 34 30 39 31 38 2f 75 73 2d 65 61 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 37 62 33 38 32 33 30 62 34 38 33 32 61 66 34 32 62 35 62 34 31 39 65 37 35 36 64 33 65 30 31 39 61 39 62 38 37 34 37 33 63 35 33 31 30 30 65 61 65 33 32 39 30 38 61 32 33 32 35 31 31 62 65 62"
    ],
    "CanonicalRequest": [
      "GET\n/1/6884/user_data_export/1/2024/09/18/03%3A35%3A48-37ef4112/export.csv\nX-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIARRX37GWVL5YUBHGV%2F20240918%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240918T033548Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&response-content-disposition=attachment%253B%2520filename%253D%2522Bloomingville%2520Daily%2520CSV%2520Inventory%2520Update.csv%2522&response-content-type=text%252Fcsv\nhost:salsify-dandelion.s3-external-1.amazonaws.com\n\nhost\nUNSIGNED-PAYLOAD"
    ],
    "CanonicalRequestBytes": [
      "47 45 54 0a 2f 31 2f 36 38 38 34 2f 75 73 65 72 5f 64 61 74 61 5f 65 78 70 6f 72 74 2f 31 2f 32 30 32 34 2f 30 39 2f 31 38 2f 30 33 25 33 41 33 35 25 33 41 34 38 2d 33 37 65 66 34 31 31 32 2f 65 78 70 6f 72 74 2e 63 73 76 0a 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 26 58 2d 41 6d 7a 2d 43 72 65 64 65 6e 74 69 61 6c 3d 41 4b 49 41 52 52 58 33 37 47 57 56 4c 35 59 55 42 48 47 56 25 32 46 32 30 32 34 30 39 31 38 25 32 46 75 73 2d 65 61 73 74 2d 31 25 32 46 73 33 25 32 46 61 77 73 34 5f 72 65 71 75 65 73 74 26 58 2d 41 6d 7a 2d 44 61 74 65 3d 32 30 32 34 30 39 31 38 54 30 33 33 35 34 38 5a 26 58 2d 41 6d 7a 2d 45 78 70 69 72 65 73 3d 36 30 34 38 30 30 26 58 2d 41 6d 7a 2d 53 69 67 6e 65 64 48 65 61 64 65 72 73 3d 68 6f 73 74 26 72 65 73 70 6f 6e 73 65 2d 63 6f 6e 74 65 6e 74 2d 64 69 73 70 6f 73 69 74 69 6f 6e 3d 61 74 74 61 63 68 6d 65 6e 74 25 32 35 33 42 25 32 35 32 30 66 69 6c 65 6e 61 6d 65 25 32 35 33 44 25 32 35 32 32 42 6c 6f 6f 6d 69 6e 67 76 69 6c 6c 65 25 32 35 32 30 44 61 69 6c 79 25 32 35 32 30 43 53 56 25 32 35 32 30 49 6e 76 65 6e 74 6f 72 79 25 32 35 32 30 55 70 64 61 74 65 2e 63 73 76 25 32 35 32 32 26 72 65 73 70 6f 6e 73 65 2d 63 6f 6e 74 65 6e 74 2d 74 79 70 65 3d 74 65 78 74 25 32 35 32 46 63 73 76 0a 68 6f 73 74 3a 73 61 6c 73 69 66 79 2d 64 61 6e 64 65 6c 69 6f 6e 2e 73 33 2d 65 78 74 65 72 6e 61 6c 2d 31 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 0a 68 6f 73 74 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44"
    ],
    "RequestId": [
      "4B9K1NKER0W61S40"
    ],
    "HostId": [
      "/m9bcWJNOsH0gBUncbzMMsYg/2SQVfXJK6uKZqmwQY3TAIZZKoTVUeI5D5wJWo9pHWQ/3PicbJY="
    ]
  }
}

I got this far by segmenting each parameter and putting it in its own request parameter:

image
image548×840 40.4 KB

Since I don’t know that API at all, I’m not sure which parts work together and what the error is telling me.

I’m attaching my blueprint in case someone sees something innocuous that I’m missing.

blueprint-amz-request.json (11.0 KB)

L

5

You could use the http get file module. It will come in as a text string but you could parse it.

If you need a csv you can use the csv app to create a csv.

6

I don’t think the problem is related to the csv file. The problem is accessing it through make. The link @Bohdan_Skovronsky shared works well if you click it in a browser. I was able to access it also using curl. But when using make, the Amazon server doesn’t give access. I suspect it’s a problem in the way make sends the parameters to the AWS server. But I’m not sure what is not correct.

L

7

That is not the issue, the issue is the link itself, that it works properly in the browser when clicked but does not work in Make nr via wget :frowning:

Thank you.