Hi all, I have been using an HTTP GET request to grab the first page of search results for a given search query. Then I am asking OPEN AI to return some basic information about the results. This has been working great for the last few months.
About 1-2 weeks ago, I started getting this error instead:
Please click here [/httpservice/retry/enablejs?sei=HTqZZ4vfBo205NoPsYaO6Ao] if you are not redirected within a few seconds.
Has anyone else experienced this?
Welcome to the Make community!
So you basically need to “visit” the site yourself to get the content. This is called Web Scraping.
However, it appears that the scraping service you are using doesn’t have JavaScript enabled, hence it is unable to fetch the results as the search engine you called displays results with JavaScript.
For web scraping, a service you can use is ScrapeNinja to get content from the page.
ScrapeNinja allows you to use jQuery-like selectors to extract content from elements by using an extractor function. ScrapeNinja also can run the page in a real web-browser, loading all the content and running the page load scripts so it closely simulates what you see, as opposed to just the raw page HTML fetched from the HTTP module.
If you want an example, take a look at Grab data from page and url - #5 by samliew
You can also use AI-powered web scraping tools like Dumpling AI.
This is probably the easiest and quickest way to set-up, because all you need to do is to describe the content that you want, instead of inspecting the element to create selectors, or having to come up with regular expression patterns.
The plus-side of this is that such services combine BOTH fetching and extracting of the data in a single module (saving operations), and doing away with the lengthy setup from the other methods.
For more information on the different methods of web scraping, see Overview of Different Web Scraping Techniques in Make 🌐
Hope this helps! Let me know if there are any further questions or issues.
— @samliew
P.S.: Investing some effort into the Make Academy will save you lots of time and frustration using Make.
It says enable javascript. This site must be using javascript that seems to be disbaled in browser settings or blocked by any extension.
Regards,
Msquare Automation - Platinum Partner of Make
@Msquare_Automation
Thank you, Sam. This is so helpful. I will explore these routes.
Do you think it is reasonable to assume that something changed on the Google end regarding Javascript?