Is it possible to scrape data from website like Reddoorz to get information or maybe the realtime information? i want to collect all the information and put it in google sheet
Welcome to the Make community!
Here is an overview of web-scraping using Make:
Traditional Web Scraping + Text Parser
If you don’t want to rely on external services which may not be free, you can always fetch the content of the page using the HTTP “Make a request” module, then use a Text Parser “Match Pattern” module to find and return the content in the source code of the page.
To do this effectively, you need to know how to setup regular expression patterns, which can get complex very quickly if you want to match multiple content around the page using a single Match Pattern module. Alternatively, you can use one Match Pattern module per content you want to extract, but this method uses more operations.
Alternatively, you can extract items using XPath using the XML “Perform XPath Query” module, but you have to use one module per extraction.
Hosted Web Scraping
If you don’t want to deal with web scraping, some apps you can use are ScrapingBee and ScrapeNinja to get content from the page.
ScrapeNinja has jQuery-like selectors in the extractor function, basically it’s how you get elements on a page. This way there are no regular expressions involved, but you can still use regex in the extractor function if you wish.
The main advantages of hosted web scraping services like ScrapeNinja is that it can handle and bypass anti-scrape measures, run the page in a real web-browser, loading all the content and running the page load scripts so it closely simulates what you see, as opposed to just the raw page HTML fetched from the HTTP module. Dedicated scraping services like these makes scraping so much more reliable, because they specialize in one thing and do it well.
If you want an example of ScrapeNinja usage, take a look at Grab data from page and url - #5 by samliew
AI-powered Web Scraping
This is probably the easiest and quickest way to set-up, because all you need to do is to describe the content that you want, instead of inspecting the element to create selectors, or having to come up with regular expression patterns.
Here is a simple example using the Dumpling AI “Extract data from URL” module:
As you can see, you can do this easily within a few seconds. Just map the URL variable, and tell it what you want extracted from the page! (You don’t even need to specify the type of data)
Hope this helps! Let me know if there are any further questions or issues.
Note: I see hundreds of posts, notifications, and messages daily on this forum, so if I missed your reply, please message me to look at your reply.
— @samliew
P.S.: Did you know, the concepts of about 70% of questions asked on this forum are already covered in the Make Academy. Investing some effort into it will save you lots of time and frustration using Make later!