YouTube search automation: how to avoid duplicates and get a limited number of videos

[AI_VY]

What is your goal?

1. Avoid duplicates and 2) Make sure that YouTube API returns limited number of results.

What is the problem & what have you tried?

I’m building an automation where I provide a list of search phrases, then for each phrase the scenario queries YouTube and retrieves up to N video IDs. After that, the videos are analyzed by IA and a summary is emailed to me.

My issue is the following:

1. In my first module (Set multiple variables), I intentionally provided 2 identical search terms. I would like to understand how I can remove duplicates later, so that any duplicate videos are filtered out. (The reason is that I’m planning to use a long list of search phrases, and if 2 different search phrases return the same video ID, I don’t want to process or analyze those videos twice). I tried to use Aggregator Array after the first HTTP module, setting the first iterator module as the source module and grouping by IDs, but it only returned the first results from each operation (each search phrase).

2. In addition, after the first HTTP > Make a request module, I added a “Limit items” module to trim the results and keep only the number of videos I need. I noticed that even when I specify the maximum number of results in the YouTube API request, sometimes the API still returns 25 results per search phrase.

Could you please explain me why this situation (extra results) happens and how to properly deduplicate video IDs in this scenario?

Thank you.

Screenshots (scenario flow, module settings, errors)

YouTube IA news.blueprint-2.json (62.7 KB)

[samliew]

Hi @AI_VY,

Could you provide a few screenshots of the issue?

[AI_VY]

Here is what I get as an output of the HTTP module:

Here are the settings of the Array aggregator:

And its output:

(I didn´t include it in the blueprint that I shared, as it didn’t give me the right result).

As for the exceeded number of results (videos IDs) when I do an API call, it doesn´t happen regularly, so I need to wait until it happens again to attach some screenshots here. It normally takes place with 1 or max 2 operations (search queries), and if it fails, it returns maximum 25 results, even if I limit it to 2, 3 or 5 (it never returns 15 or 20 results).

[samliew]

@AI_VY

Here are the settings of the Array aggregator:

That’s what happens when Group By is used. You’ll need to use a Set Variable module to extract the first item in the array, if that’s what you want.

Alternatively, you can avoid using Group By, then, you can use this built-in function in a Set Variable module after the aggregator:

• distinct
  distinct(array; [key])

Here’s an example of how you can use this function:

{{ distinct(86.data.items; id.videoId) }}
_{(copy-paste the above into the field, or type it exactly as shown)}

Hope this helps! If you are still having trouble, please provide more details.

— @samliew

[AI_VY]

Thanks a lot for your help. I think I didn’t explain myself clearly. I’m not trying to just pick the first item of each operation.

Let me clarify with a concrete example.

I have 3 search phrases, 2 of which are intentionally duplicated: phrase 1, phrase 2, phrase 3.

For each phrase I fetch up to 2 YouTube videos:

• Phrase 1: results A1, A2

• Phrase 2 (same as phrase 1): results A1, A2

• Phrase 3: results B1, B2

What I need is to end up with one single array that contains unique video IDs: A1, A2, B1, B2.

YouTube IA news.blueprint-3.json (119.1 KB)

[samliew]

To do this, you can use these built-in functions:

• map
  map(complex array; key; [filter by key]; [csv-values for filtering])

• flatten
  flatten(array)

• deduplicate
  deduplicate(array)

Here’s an example of how you can use these functions together:

{{ deduplicate(flatten(map(array; key))) }}

e.g.:

{{ deduplicate(flatten(map(132.array; "data.items")); "id.videoId") }}

Oh, and you don’t use the GroupBy option in the aggregator, because you are aggregating arrays, not bundles.

Module Export - quick import into your scenario

You can copy and paste this module export into your scenario. This will import the modules (with fields/settings/filters) shown in my screenshots above.

1. Move your mouse over the line of code below. Copy the JSON by clicking the copy button on the right of the code, which looks like this:
   

2. Enter your scenario editor. Press ESC to close any dialogs. Press CTRLV (paste keyboard shortcut for Windows) to paste directly in the editor.

3. Click on each imported module and re-save it for validation. There may be some errors prompting you to remap some variables and connections.

JSON module export — paste this directly in your scenario

{"subflows":[{"flow":[{"id":133,"module":"util:SetVariables","version":1,"parameters":{},"mapper":{"variables":[{"name":"mapped","value":"{{map(128.array; \"data.items\")}}"},{"name":"flattened","value":"{{flatten(map(128.array; \"data.items\"))}}"},{"name":"deduped","value":"{{deduplicate(flatten(map(128.array; \"data.items\")); \"id.videoId\")}}"}],"scope":"roundtrip"},"metadata":{"designer":{"x":1781,"y":353}}}]}],"metadata":{"version":1}}

Hope this helps! If you are still having trouble, please provide more details.

— @samliew

[AI_VY]

Great. Thank you so much for such a detailed and visual explanation. The Set Multiple Variables module helped me a lot to understand how the above mentioned functions work for a complex data structure.

And, actually, it was a great discovery for me that we can call the data as “data.items” (before I thought that we should call the directly as “items”).

[samliew]

No problem, glad I could help with "YouTube search automation: how to avoid duplicates and get a limited number of videos"!

I’d recommend going through the Make Academy if you haven’t yet!

Here are some useful links and guides you can use to learn more on how to use the Make platform, apps, and app modules. I found these useful when I was learning Make, and hope they might benefit you too —

Learn Make

• Help Centre | Tutorials – Make’s “manual” - search for help here first
• Make Academy – Basics 101: Learn Make properly to get your money’s worth
• Mapping – What is mapping? What can I map?
• Mapping with arrays – How to map items in an array
• Date Format: tokens for parseDate | tokens for formatDate
• HTTP modules – Make a request, Get (download) a file
• Webhooks – Error Handling, Responding to webhooks

How-Tos

• Router Magic Formula - YouTube
• Error Handlers in Make - YouTube playlist
• Aggregate an array for mapping complex fields
• Getting started with OpenAI - How to setup and use OpenAI in Make
• Text Parser module - Use pattern matching to extract the text you want
• Webhooks – Make Academy - tutorial on how to use Webhooks

Useful to Know

• How to schedule scenario for first Monday next month (recurring)
• How to schedule scenario for the last day next month (recurring)
• How to Extract URL Query Parameters Into Variables
• Overview of Different Web Scraping Techniques in Make
• [Free] Use Groq AI to filter spam emails and form submissions!
• Automate Follow-Up of Email Leads a Week After
• How to group bundles in X amount of bundles (Chunking Arrays)

My Custom App — Absolutely Free!

Save development time and operation credits with these utility modules: Chunk Text; Chunk Array; Chunk Words; Multiple Find & Replace; Collection to Array/String List; Execute JavaScript; Estimate Tokens; Calculate Difference Between Two Dates; Get Next Business Day (with holidays); List Upcoming Dates of Day of Week; and more!

If you have a follow-up question, please start a new thread. Thanks!

— @samliew