I am having trouble getting Chatgpt to read a webpage

If you’re fetching the article’s URL, you’ll probably need to do some kind of HTML sanitisation, or convert from HTML to text, or both, before you can pass it to OpenAI.

There is no right way of doing it. You can even use a web scraping module to simplify the job.

For web scraping, some apps you can use are ScrapingBee and ScrapeNinja to get content from the page.

I’ve used ScrapeNinja, and you can use jQuery-like selectors there in the extractor function.

ScrapeNinja also can run the page in a web-browser so it closely emulates what users see, as opposed to just the raw page HTML fetched from the HTTP module.

If you want an example, take a look at Grab data from page and url - #5 by samliew

samliewrequest private consultation

Join the Make unofficial Discord server!

1 Like