Scrape and perform actions on a web page

Features
1

Hi there!

I want Make to be able to log in to my utility companies’ website and download a PDF of my latest bill.

Is it possible for Make to run some type of macro on a website? There is no external API unfortunately, so I’m limited to scraping their site, looking for keywords like Account number, “View Bill” etc, then I would like Make to “click” on “View Bill” and retrieve the PDF. They use Basic Auth, so there should not be any issue there…

A lot to ask, I know, but I’m hoping someone has tried this before!

2

If your utility company has the option to receive the bills electronically as PDFs via email, that’s your best option.

I have my scenario set up to watch for new incoming utility bills via email, upload a copy to Google Drive and update a Google Sheet with the values.

If that’s not possible, consider changing utility providers or requesting that bills be sent to your email.

2 Likes
3

The logging-in is the tricky bit because we’re not sure whether there is any CORS and CSRF protection, or anti-scraping mechanisms in place (e.g.: Cloudflare). Also, the server doing the logging-in will need to store a copy of the login cookie, which is not doable using just the HTTP modules in Make.

In layman terms, what one would need to do is write a Node.js (or any server-side) app to “copy” the network requests from your web browser that are sent to the server. To make a copy of the network requests you’ll need to know how to use the browser’s developer tools to create a script that does the same thing. Then you’ll host the script online (Google Cloud/AWS/Heroku/etc.) to either run on a cron schedule and call your scenario’s webhook, or your scenario makes a HTTP request to your script to make the requests, and return the file.

2 Likes