Sub-iterating through all bundle collections / flatten array aggregator return

[jay.serquina]

Is there a way to flatten the array aggregator return, or sub-iterate through the bundle collections, so I get my un-nested array of collections?

---

Context:
I’m parsing through an ATS API return, that has one or more nested Collections per data row.

{ response: { data: [ [ {collection1}, {collection2} ] ] ] } }
{ response: { data: [ [ {collection3} ] ] } }
{ response: { data: [ [ {collection4}, {collection5} ] } }

Expectation:
I’m using an iterator-array aggregator, to parse through the collections, make a transform, and then list all of the collections as an array one level deep.

[ {collection1}, {collection2}, {collection3}, {collection4}, {collection5} ]

Reality:
Unfortunately, the array coming out of the array aggregator returns it like so:

Bundle 1: [ {collection1}, {collection2} ]
Bundle 2: [ {collection3} ]
Bundle 3: [ {collection4}, {collection5} ]

…which is frankly very frustrating (and I hope I’m reasonable with thinking that an iterator, well, iterates)

Steps Tried
Thinking I needed to iterate again, I tried adding another iterator-array aggregator after the first one, but unfortunately it would return the same result as the first iterator-array aggregator group.

Ideally I don’t want the solution to get too fancy or highfalutin, since my scenarios are peer-reviewed by our architecture and BI teams, who may not understand why I would use funky workarounds.

Scenario layout

[samliew]

Welcome to the Make community!

Have you tried using the built-in function flatten on the top-level data array? I think it should be as simple as that - give it a try with a Set Variable module.

[jay.serquina]

Thanks, @samliew, glad to be here!

I should have clarified that there’s no single top-level data array, instead it’s a whole slew of responses like this:

{ response: { data: [ [ {collection1}, {collection2} ] ] ] } }
{ response: { data: [ [ {collection3} ] ] } }
{ response: { data: [ [ {collection4}, {collection5} ] } }

I’ve edited the OP to reflect this.

[samliew]

Just a word of caution, I’ve never seen 4000+ operations used in just a single scenario run.

You might want to watch out for that otherwise you’ll find a huge bill.

We usually aim to use about 30-50 ops per scenario on average.

Try aggregating your results into a single array so they don’t run the modules down the line so many times!

Every result (item/record) from a search/match module will output a bundle. To “combine” them into a single structure, you’ll need to use an aggregator of some sort.

Aggregators are modules that accumulate multiple bundles into one single bundle. An example of a commonly-used aggregator module is the Array aggregator module.

[samliew]

@jay.serquina

I should have clarified that there’s no single top-level data array, instead it’s a whole slew of responses like this

You might want to start describing your scenario from earlier up the chain then. You might be doing something terribly wrong if you’re not aggregating the bundles somehow.

[jay.serquina]

This is a pretty clunky ATS, which unfortunately their search API is set up to serve URIs of results in batches of 1000 rows, whether it has relevant data or not I even have a filter between modules to scrub blank returns

But that’s a different discussion, and kudos if you know which ATS this is lol

The fact that it’s returning the results as an API URI per data row, is a pain

The scenario is marked to run on-demand only, but while debugging I have no choice but to run the thousand API calls

[samliew]

Let me think about this for a bit. Maybe there are other experts on this forum who may be able to come up with a better way of doing what you’re trying to do.

But I’m certain making the 1000 HTTP requests in Make is not a really good idea (unless you have an unlimited budget), so think of some way to “farm” that out to an external service? You can possibly run some custom JavaScript to make those HTTP fetches and combine/format the data in a single operation, for example.

[jay.serquina]

For the here and now, do I want to use flatten after the array aggregator into a JSON, and then iterate from there? And how best should I do it?

Also, I can make another Q/A post here over the crazy amount of requests. Our account executive already knows about our predicament, but I would love to have a community thread to show him about it.

[jay.serquina]

Update: flattening doesn’t work with my flow; while each Collection in the bundle would get array-flattened, I could not find a straightforward way to merge all of the Collections in the bundles into a “super-array” and then run flatten on that

Any result coming out of the array aggregator would still show one or more Collections per Bundle instead

Is this a Make limitation? @samliew

[samliew]

Can I see a screenshot of the output bundles of the HTTP module [12]? More screenshots of what you currently have, especially mappings, and actual data, would help.

(You don’t have to rerun the scenario, use the scenario History tab to access previous runs)

I think you might be able to combine them all with a text aggregator, so that modules down the line only use singular operations.

If you could provide example output bundles of the HTTP module by running the scenario, then click the white speech bubble on the top-right of each module and select “Download output bundles”.

A.

Save the bundle contents in your text editor, save, and upload it here into this discussion thread, or upload to Google Drive and provide a share link.

Uploading files here will look like this:

bundle.txt (12.34 KB)

B.

If you are unable to upload files on this forum, alternatively you can paste the formatted output bundle in this manner:

• Either add three backticks ``` before and after the code, like this:

  ```
  <sup>input/output bundle content goes here</sup>
  ```

• Or use the format code button in the editor:
  

Providing the output bundles will allow others to replicate what is going on in the scenario even if they do not use the external service.

This will allow others to better assist you. Thanks!