Get File from URL: JSONL format

[Zii]

Hi All, I’m stuck and need some help.

I run a HTTP Get a File scenario and get a file in the JSONL format. I want convert this to a CSV and save it in my Google Drive.

The data from the file is in a similar format as below.

{"id":"gid://shopify/Product/1921569226808"}
{"id":"gid://shopify/ProductVariant/19435458986123","title":"52","__parentId":"gid://shopify/Product/1921569226808"}
{"id":"gid://shopify/ProductVariant/19435458986040","title":"70","__parentId":"gid://shopify/Product/1921569226808"}
{"id":"gid://shopify/Product/1921569259576"}
{"id":"gid://shopify/ProductVariant/19435459018808","title":"34","__parentId":"gid://shopify/Product/1921569259576"}
{"id":"gid://shopify/Product/1921569292344"}
{"id":"gid://shopify/ProductVariant/19435459051576","title":"Default Title","__parentId":"gid://shopify/Product/1921569292344"}
{"id":"gid://shopify/Product/1921569325112"}
{"id":"gid://shopify/ProductVariant/19435459084344","title":"36","__parentId":"gid://shopify/Product/1921569325112"}
{"id":"gid://shopify/Product/1921569357880"}
{"id":"gid://shopify/ProductVariant/19435459117112","title":"47","__parentId":"gid://shopify/Product/1921569357880"}

When I try to parse the JSON I get an error " * Buffer can’t be converted to text for parameter ‘json’."

Here is what my scenario looks like:

Here is the data output from the Get a File Module:

Where am I going wrong?

If I remove the Parse JSON module and connect it to Create CSV module I am able to see the data:

I want to split this into two columns with the current sample data: ID and Title. How can I achieve this?

[samliew]

What’s the error in your JSON module?

I was able to do it this way:

Output

[Zii]

Hi @samliew

This is how I set it up:

and this is the error I get:

[samliew]

I know what’s the problem now, the encoding issue when downloading certain files.

I have mentioned a possible workaround for this here: Read text in .txt file stored in google drive - #4 by samliew

You can use a “Convert encoding” module using the same input and output encoding, then you can use the output of this module in your Parse CSV module.

Alternatively, you can simply try   {{toString(6.data)}}   in your JSON string field.

[Zii]

Hi @samliew, thank you for your assistance.

With both suggestions, I get an error saying JSON format is not valid. I am assuming because the original file is in JSONL format it’s not recognizing it?

[samliew]

@Zii

I get an error saying JSON format is not valid

Ah yes, JSON and JSONL formats are not interchangeable, I’m afraid. You can’t use a JSON module to directly parse JSONL.

You need to split the JSONL by newlines, then use an Iterator before Parse JSON on each record.

[Zii]

Hi @samliew, sorry for the delay in getting back to you. I’m not entirely sure how to implement your suggestion. Any chance you can give me some pointers to how I can split it into new lines and then use iterator?

[JimLarrack]

I’m still trying to sort this out as well did anyone figure it out? I’m able to use a pattern match (?<=\n|^){.+?}(?=\n|$) and then a parse JSON to get the collections formatted, but it will have to run for each collection which would be a lot considering i’m using the bulk operation api from shopify. Ideally we wouldn’t run parse json for hundreds if not 1,000 results to format the collections correctly

[Joseph_Accountant]

@JimLarrack & @Zii - I may be able to help out - this is my solution when I had a JSONL data returned in one of my modules.

Here is the overview, and I will provide screenshots below on each module.

toString() will transform the binary data into text, using split() on ‘newline’ splits the JSONL into individual JSON in an array, and the iterate module will split out each JSON into a bundle.

The Text Aggregator function is used to combine the JSON bundles back into a single JSON, it’s important to use the , separator to ensure valid JSON is the result.

Last, use the Parse JSON module to parse our now valid JSON into usable bundles of code (or leave unparsed depending on your needs of course). Note that I added around the JSON to make it valid.

I hope this helps,
-Joe

[JimLarrack]

Hi Joe, thanks for the reply!

Nice I think this actually works better. I figured it out w/ chat GPTs help, basically ran an a couple text parsers so that the parse json would recognize the data, but its not very pretty haha

Thank you!