How to deduplicate with iterator?

[Peter_Jivotovski]

I’m iterating multiple URLs through an HTTP module to retrieve the resolved URLs, then I’m putting those into an array, then iterating through them again to get rid of duplicates. However, I’m having trouble getting rid of the duplicates—I do not understand at all how the deduplicate & distinct functions work.

Could someone please explain what I should put into the iterator and why?

[Prem_Patel]

Hi @Peter_Jivotovski Welcome to the community.

Looks like everything is perfect. Just need to make 1 change in source module in array aggregator, module no. 5. Change it with module which is giving combined bundles before http module in single operation, which are further processed as solo operation in http module. so aggregator can combine all url correctly.

Let me know if you need more help!

Best,
@Prem_Patel

[Peter_Jivotovski]

@Prem_Patel

Looks like everything is perfect. Just need to make 1 change in source module in array aggregator, module no. 5. Change it with module which is giving combined bundles before http module in single operation, which are further processed as solo operation in http module. so aggregator can combine all url correctly.

I’m sorry I don’t understand at all. Can you tell me exactly what nodes I should have and in what order?

Also, I’ve read that deduplicate might not work here and I would have to use distinct. Is that not true?

[Prem_Patel]

Here is a sample workflow :

Aggregator.blueprint.json (7.2 KB)

Instruction to import above blueprint : Create a New scenario > Use import Blueprint option

Once imported, Save scenario, do a test run and refer instructions shared below.

Now look into the aggregator module :

Choose Source Module accordingly which gives output of your desired URL in 1 operation. My example will return 3 URL bundles in 1 operation.

Http module will process each module 1 by 1 in 3 different operations

Aggregator module combines allresolved URL in 1 array.

In this example as i am using same values to use deduplicate function in iterator.

And output after that is shown below.

Now this whole explanation will help you in deduplicating if you use aggregator with correct source module.

Best,
@Prem_Patel

[Peter_Jivotovski]

It worked, thank you so much

[Prem_Patel]

Glad it worked as always! @Peter_Jivotovski

Please feel free to mark the earlier reply as the solution

Best,
@Prem_Patel