How to convert one big JSON into multiple smaller JSON files?

[soft.beach7708]

I’m pulling conversations from a SLACK channel, and I’m running into several problems trying to break it down into multiple JSON files to work them out individually because it’s too much text.

1. First, for some reason, the Slack API brings duplicated messages in the output of the conversation, so I need to later clean this, but this causes difficulty in grouping this data, because I don’t know if a lot of the duplicated data is from my mistakes in the scenario or from the source.

2. I only need the “user”, “ts” and “text” from the [Messages] array, all else I can discard.

I added this to the text aggregator, but then I’m not sure how to build the JSON after the repeater.

3. The Create JSON module has the 3 input boxes mapped: user, ts, and messages but I don’t know how to map the variables.

I tried adding from the HTTP module, but it’s just duplicated output

I attached the JSON output that I’m trying to break down.

test-output-slack.json (90.7 KB)

[samliew]

You can map the messages array in the Iterator, then use an Array Aggregator (not Text Aggregator) to select the fields you want.

[soft.beach7708]

Thanks, what do I need to add to the Repeater & Set Variable so it breaks down the array?

I tried just adding the array variable, but a lot of text goes missing in the final JSONs

Added the Array Aggregator

Selected the 3 variables

< No idea how to break down this array into chunks >

Added the array variables to the JSON creator module.

[soft.beach7708]

Any other ideas? …

[soft.beach7708]

I still haven’t been able to solve this fully, the issue is that when I try to break down the JSON either (A) important text goes missing (too little text) OR (B) a lot of text gets repeated in different chunks (still too much text).

I’ll try a bit more before posting the solution I came up with, but still not ideal, essentially I kept the repeated text on each chunk, and I then sent each one to a google sheet row, and I ran a custom Google Sheet Script to find and clean duplicated texts.

[samliew]

Welcome to the Make community!

@samliew

You can map the messages array in the Iterator, then use an Array Aggregator (not Text Aggregator) to select the fields you want.

Like I said earlier, this can be done this way. You’ll need a minimum of three modules:

1. Iterate messages array

2. Aggregate, selecting only the three fields you want (28 items)

3. Use built-in distinct function (26 items)

Give it a go and let us know if you have any issues!

[samliew]

If you need the output to be JSON, insert the messages_unique_texts variable into a “Transform to JSON” module.