I have an API which scrapes the serper dev and paginates it until the result is empty. The problem is that serper dev has pagination in which case i want to iterate the page until i get the array result as empty then i will stop scraping. I want also that all the results from different pages will be saved as one array then be sent to webhook response.
Welcome to the Make community!
If the API returns the number of pages in the API, or the total number of items, you should be able to use the Flow Control “Repeater” module.
For more information, see https://www.make.com/en/integrations/builtin, and https://www.make.com/en/help/app/builtin in the help centre.
Hope this helps! Let me know if there are any further questions or issues.
— @samliew
P.S.: Investing some effort into the Make Academy will save you lots of time and frustration using Make.
It doesnt return the number of pages unfornutaly so i really need to loop every page and save every result of the page until the result is empty. So for example, in every page there are 10 results, and you wait until a page is empty then the scraping will stop and i should save all the result from different pages and send it back using webhook response.
What about the total number of items and page size?
For more information, see Master Pagination in Make – Retrieve All Data from APIs
Hope this helps! Let me know if there are any further questions or issues.
— @samliew
P.S.: Investing some effort into the Make Academy will save you lots of time and frustration using Make.
There is also no total number of results. like if you query something for page 1, it will just give you 10 results per page. then you need to go to http request again for page 2 to get another 10 results until there is no more result for a certain page.
Example
Here is an example of how your scenario could look:
This is just an example. Your final solution may or may not look like this depending on your requirements.
Hope this helps! Let me know if there are any further questions or issues.
— @samliew
P.S.: Investing some effort into the Make Academy will save you lots of time and frustration using Make.
Can you please check the workflow i sent here. I’m having a problem because since i have a repeater for example i set it from 1 to 20. then it will have 20 operations but my problem is that for webhook response, only the operation 1 is being sent back to clay. How can i do that all the bundles from different operations will be stored first then i’ll send them all to webhook response.
Setting the Correct Aggregator Source
You need to set the “Source Module” field of the aggregator to where the bundles are coming from. This is usually an Iterator module, but can also be a search/list/repeater module.
For more information, please refer to the Make Academy.
Combining Bundles Using Aggregators
Every result (item/record) from trigger/iterator/list/search/match modules will output a bundle. This can result in multiple bundles, which then trigger multiple operations in future modules (one operation per bundle). To “combine” multiple bundles into a single variable, you’ll need to use an aggregator of some sort.
Aggregators are modules that accumulate multiple bundles into one single bundle. An example of a commonly-used aggregator module is the Array aggregator module. The next popular aggregator is the Text Aggregator which is very flexible and can apply to many use-cases like building of JSON, CSV, HTML.
You can find out more about the other types of aggregator modules here:
- Array aggregator
- Text aggregator
- Numeric aggregator
- Table aggregator
- JSON aggregator
- CSV aggregator
- Archive (ZIP) aggregator
- other app-specific aggregators
Hope this helps! Let me know if there are any further questions or issues.
— @samliew
P.S.: Investing some effort into the Make Academy will save you lots of time and frustration using Make.
This is my current scenario. As you can see my problem is that during the initial loop, it will pass the result of the operation 1 quickly to the webhook response so meaning the remaining operations will not be sent anymore to clay.
See https://academy.make.com/courses/IntermediateC01
Hope this helps! Let me know if there are any further questions or issues.
— @samliew
P.S.: Investing some effort into the Make Academy will save you lots of time and frustration using Make.
So i need to switch the place of array aggregator and the repeater? I used the repeater to as the iterator for my page because i used it like this:
No. Please see the answer two posts up about the Aggregator Source.
Hope this helps! Let me know if there are any further questions or issues.
— @samliew
P.S.: Investing some effort into the Make Academy will save you lots of time and frustration using Make.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.