Hi 
I’ve set up a web scraper scenario. It works pretty well.
The problem is I can only scrape the first page. How do I scrape the other pages???
The scenario is triggered with an HTTP Make a request, which only takes one URL.
Is there a way to do this in Make? Preferably without another paid tool like Dumpling AI?
Welcome to the Make community!
For further assistance, please provide the following:
Please export the scenario blueprint file to allow others to view the mapped variables in the module fields. At the bottom of the scenario editor, you can click on the three dots to find the Export Blueprint menu item.
Please provide the output bundles of the modules by running the scenario (or get from the scenario History tab), then click the white speech bubble on the top-right of each module and select “Download input/output bundles”.
Save each bundle contents in your text editor as a bundle.txt file, and upload it here into this discussion thread.
If you are unable to upload files on this forum, alternatively you can paste the formatted bundles in this manner:
Here are two ways to format text so that it won’t be changed by the forum:
A. Type code block manually
Add three backticks ``` before and after the content/bundle, like this:
```
content goes here
```
B. Highlight and click the format button in the editor
Providing the input/output bundles will allow others to replicate what is going on in the scenario even if they do not use the external service.
Following these steps will allow others to assist you here. Thanks!
Hi @samliew
Here is the blueprint (attached)
Do you want ALL the bundles? There are 19 modules, so I need to confirm that’s what you want before I download them and put them in .txt files.
Tchau M
blueprint.json (128.5 KB)
Here are some closer screenshots of the scenario:
I can’t find much information on Make.com about pagination. Does anyone have a good resource I can use?
Hmm, looks like the site you are scraping uses GraphQL and cursors, instead of page numbers.
The easiest way is probably to use ScrapeNinja in that case, as it can click on the load more button to get more items.
Hope this helps! Let me know if there are any further questions or issues.
— @samliew
P.S.: Investing some effort into the Make Academy will save you lots of time and frustration using Make.
Thanks, I’ll check out ScrapeNinja.
They have a free plan, so I can test if it works before committing.
Hi @samliew can you show me how to configure ScrapeNinja-- the information on their site is not good.
I’ve set up the RapidAPI key and tried it in a Make scenario, but I get a 404 error.
I can’t even find useful tutorials on YouTube for ScrapeNinja.
You can use the ScrapeNinja modules instead.
Hope this helps! Let me know if there are any further questions or issues.
— @samliew
P.S.: Investing some effort into the Make Academy will save you lots of time and frustration using Make.
