How to combine the results of multiple operations into one single array and avoid duplicates

Questions
,
1

:bullseye: What is your goal?

To get 1 single array of video IDs, avoiding duplicates

:thinking: What is the problem & what have you tried?

I’m building an automation where I provide a list of search phrases, then for each phrase the scenario queries YouTube and retrieves up to N video IDs. After that, the videos are analyzed by IA and a summary is emailed to me.

In my first module (Set multiple variables), I intentionally provided 2 identical search terms. I would like to understand how I can remove duplicates later, so that any duplicate videos are filtered out. (The reason is that I’m planning to use a long list of search phrases, and if 2 different search phrases return the same video ID, I don’t want to process or analyze those videos twice).

Let me explain it on the example:
I have 3 search phrases, 2 of which are intentionally duplicated: phrase 1, phrase 2, phrase 3.
For each phrase I fetch up to 2 YouTube videos:

  • Phrase 1: results A1, A2
  • Phrase 2 (same as phrase 1): results A1, A2
  • Phrase 3: results B1, B2

What I need is to end up with one single array that contains unique video IDs: A1, A2, B1, B2, and ideally in 1 operation.

I tried to use Aggregator Array after the first HTTP module, setting the first iterator module as the source module and grouping by IDs, but it only returned the first results from each operation (each search phrase). Distinct() function also didn´t give me the expected result.

:camera_with_flash: Screenshots (scenario flow, module settings, errors)

YouTube IA news.blueprint-3.json (119 KB)

3

After all YouTube search modules, use Array aggregator to collect all video IDs, then in a Set variable use {{distinct(map(aggrega ted Array; "id"))}} to get a single array of unique video IDs only.

4

Hello,

You already have created a topic about this previously here,

:link: YouTube search automation: how to avoid duplicates and get a limited number of videos

Please avoid creating duplicate threads so we can keep this forum organized, and give everybody a fair chance to get their questions answered.

For more information about using this forum, see:

Thanks for your understanding!

1 Like