Process each document in a folder with AI (conditional summarization), then aggregate all summaries into a single output

[ProQuest]

What is your goal?

From a folder, retrieve documents and process them one by one using an iterator. For each document I get its content, set variables (text and length), and then use a router to split based on document length (to avoid OpenAI token limits). For now, I’m focusing on the short path where the document is summarized using OpenAI.

At this point everything works correctly and I get one summary per document (per bundle).

The problem is at the end. I want to aggregate all OpenAI summaries into one combined output and then continue the flow with that single result.

However, the Text Aggregator runs once per bundle instead of combining everything, so I never get one final merged output.

I tried placing the Text Aggregator after OpenAI, changing the source module, removing the router, and aggregating after setting variables, but it still behaves like it’s inside the iteration.

What I’m trying to understand:
Where exactly should the Text Aggregator be placed in this kind of flow?
What should be set as the source module (Iterator or OpenAI)?
How can I make sure aggregation happens after all bundles are processed and not per bundle?
Does the router break aggregation and do I need to merge paths before aggregating?

The ideal flow I want is:
Get files → Iterator → Get content → Set variables → Router (short/long) → OpenAI (summarize) → aggregate all summaries → continue with one combined result.

What is the problem & what have you tried?

As of now, after the AI step I implemented a Text Aggregator (source: Iterator), but it still runs for each individual bundle. As a result, it does not aggregate the results — the number of operations equals the number of bundles.

Secondly, even if this worked, I want to have a third path that runs only once after all documents in the iteration are processed, not per bundle.

(By the way, the OpenAI warning is just because I force-stopped the scenario.)

Screenshots (scenario flow, module settings, errors)

[Stoyan_Vatov]

Hey there,

your screenshot is low res and I can’t really see the bundles each module produces or the names of the specific modules you are using, but I will take a wild guess here.

Iterator modules accept arrays as inputs and produce separate bundles per item inside the array. Search modules already return separate bundles per item found. So iterating the result of a search module doesn’t actually do anything.

Aggregators combine bundles together yes, but the source module is key. The source module should be a module that produced extra bundles. You can check that by comparing how many times a module ran vs how many times the next module after it ran.

I guess in your case that would be the google drive module and not the iterator and you should change your source module to that one.

Or, please provide a cleaner picture so we can see if this is indeed the case, or something else is going on.

[samliew]

Looks like Search Files/Folders is returning more than one bundle. You do not need to iterate anything coming out of this module.

If you are expecting two files (bundles), simply delete the Iterator module, and point the Aggregator at the Search Files module.

[ProQuest]

Thanks both of you ! That is correct Search is already creating bundles! Its working!

@Stoyan_Vatov @samliew