I am running into an issue for a timeout with a valid URL: * https://www.tga.gov.au/resources/consultation/consultation-updates-australian-medicine-labelling-rules-support-medicine-safety
I calling the website via the HTTP Request module. I can visit the website, yet when I add the website to the HTTP Request module, I keep getting a timeout stating that the Origin is “Make” (see image). Anyone have any clue why I can’t call this URL?
Here is the input:
This is because the server has anti-scrape measures.
Try other scraping methods described in this article.
So you basically need to “visit” the site yourself to get the content. This is called Web Scraping.
For web scraping, a service you can use is ScrapeNinja to get content from the page.
I’ve used ScrapeNinja, and you can use jQuery-like selectors in the extractor function.
ScrapeNinja also can run the page in a real web-browser, loading all the content and running the page load scripts so it closely simulates what you see, as opposed to just the raw page HTML fetched from the HTTP module.
If you want an example, take a look at Grab data from page and url - #5 by samliew
For more information on the different methods of web scraping, see Overview of Different Web Scraping Techniques in Make 🌐
Hope this helps! Let me know if there are any further questions or issues.
— @samliew
P.S.: Investing some effort into the Make Academy will save you lots of time and frustration using Make.
