A web scraping method to scrape many websites of different structure and content

Questions
21

@Juliusforster

Thank you for the answer to the first question :smiley: The flow will make requests to thousands of websites in my Google Sheet. But I guess since these websites are all separate from each other, it shouldn’t be a problem. It’s only when you make requests to the same website domain (i.e. within the web pages in the sitemap.xml) that you need to be careful. But as you said, you most likely will only make 5-10 web page requests within the same domain. :slight_smile:

I’m aware of Apify, but didn’t realize there’s a module of Apify within Make.com. That’s good. So I guess I can use Appify to solve the worry of not needing to add Headers, Delays and Captcha measures, right?

22

I don’t even know if there’s a Make.com Module for apify. I usually just use the API.

23

@Juliusforster

2- For question two, I tried referencing the ‘Ice-breaker’ column in the ‘Update a Cell’ module

image
image1916×1590 179 KB

But this did not work. I’m not sure if this is what you meant for me to do, in your answer. So if I misunderstood.

image
image2844×1168 401 KB

24

That’s the column, not the row

25

Thanks @Juliusforster , I managed to work out the ‘Update a Cell’ task. I’ll mark your original response with the video as the solution. Thank you so much once again for your help. :slightly_smiling_face: :pray: :pray: :pray:

26

Thanks for the video & the blueprint! I am building a similar automation, so this video was exactly where I was stuck. However, the /sitemap.xml you shared does not work on every site for me. Some are sitemap-index.xml, some are page-sitemap.xml etc. So right now I got it working with around 50% of the websites.

Can you make an update video on this? :slight_smile: Would be really helpful!

27

Hi @Dean1, happy I could help you with the video as well :smiley:
You are right, although from my experience around 90% of sites have the sitemap under the path /sitemap.xml.

I would probably use a router as well as a filter to check if the sitemap has been found (or if it returned xml payload). If the sitemap hasn’t been found, it tries with a different one.

You could check:
/sitemap-index.xml
/sitemap.php
/sitemap.txt
/sitemap.xml.gz
/sitemap/
/sitemap/sitemap.xml
/sitemapindex.xml
/sitemap/index.xml
/sitemap1.xml
/sitemap_index.xml

The sitemaps with index in the url are sitemaps that give you an overview over all other sitemaps for the website.

Hope that helps :party_blob: