Has anyone experience in scraping Google Jobs? I’ve tried but cannot make it work properly. Is it even possible to get data back with make.com on Google Jobs?
What are you using to scrape the site? Google blocks scraping so you need to use one of the webscraping apps. And even that is not always gonna be 100%
okay thanks. We were not using one the the scraping apps.
Welcome to the Make community!
So you basically need to “visit” the site yourself to get the content. This is called Web Scraping.
Web Scraping
For web scraping, some apps you can use are ScrapingBee and ScrapeNinja to get content from the page.
I’ve used ScrapeNinja, and you can use jQuery-like selectors in the extractor function.
ScrapeNinja also can run the page in a real web-browser, loading all the content and running the page load scripts so it closely simulates what you see, as opposed to just the raw page HTML fetched from the HTTP module.
If you want an example, take a look at Grab data from page and url - #5 by samliew
AI-powered “easier” method
You can also use AI-powered web scraping tools like Dumpling AI.
This is probably the easiest and quickest way to set-up, because all you need to do is to describe the content that you want, instead of inspecting the element to create selectors, or having to come up with regular expression patterns.
The plus-side of this is that such services combine BOTH fetching and extracting of the data in a single module (saving operations), and doing away with the lengthy setup from the other methods.
For more information on the different methods of web scraping, see Overview of Different Web Scraping Techniques in Make 🌐
Hope this helps! Let me know if there are any further questions or issues.
— @samliew
P.S.: Did you know, the concepts of about 70% of questions asked on this forum are already covered in the Make Academy. Investing some effort into it will save you lots of time and frustration using Make later!
If the above methods don’t work, you can find a scraping API on RapidAPI.
I found some that scrape Google Jobs:
How to call an API on RapidAPI
Use the HTTP “Make an API Key Auth Request” module.
Create a new keychain connection and insert your RapidAPI API Key.
Key: <YOUR_RAPIDAPI_KEY>
API Key parameter name: X-RapidAPI-Key
You can reuse this RapidAPI keychain for all API calls to RapidAPI – you’ll just need to change the X-RapidAPI-Host value based on the API you are calling.
Hope this helps! Let me know if there are any further questions or issues.
— @samliew
P.S.: Did you know, the concepts of about 70% of questions asked on this forum are already covered in the Make Academy. Investing some effort into it will save you lots of time and frustration using Make later!
1 Like
Thank you for the comprehensive answer. Scraping in general already works, we were only concerned with the special combination of make.com and Google Jobs.