Hello there,
I want to create an integration in which I automatically check the RSS field of the OpenAI Blog, than scrape the content via the URL with the tool ScrapingBee.
Unfortunately I get the following error:
" RuntimeError
[400]
Origin
ScrapingBee
Does anyone encountered a similar problem or knows how to fix it?
Thank you in advance!
With best regards,
Felix
I believe OpenAI has implemented anti-scrape measures on their website and blog.
https://openai.com/blog/chatgpt-can-now-see-hear-and-speak
I would use ScrapeNinja “Scrape (Real browser)” to emulate a real person visiting the site using a web browser, as client-side JavaScript needs to run to retrieve the data.
You CANNOT use normal scraping apps like ScrapingBee, or HTTP “Make a request” module to fetch this data.
I didn’t know that :0
Thank you for your answer! I will try out ScapeNinja with hopefully positive results!!
Have a great day!
