Scrape Content from Redirected Google News Links

How To
1

I read an advise about getting HTML and then extract a link from there with a text-parser, but in html that i get from RSS-module linked to google news i do not have a direct link. Can this problem be solved? How can i retrieve a direct link from RSS-module linked to google news to scrape the website further? Thanks in advance

3

Welcome to the Make community!

To allow others to assist you with your scenario, please provide the following:

1. Relevant Screenshots

Please share screenshots of your scenario, any error messages, relevant module fields, and filters in question? It would really help other community members to see what you’re looking at.

You can upload images here using the Upload icon in the text editor:

2. Scenario Blueprint

Please export the scenario blueprint file to allow others to view the mapped variables in the module fields. At the bottom of the scenario editor, you can click on the three dots to find the Export Blueprint menu item.

3. Output Bundles of Modules

Please provide the output bundles of the modules by running the scenario (or get from the scenario History tab), then click the white speech bubble on the top-right of each module and select “Download input/output bundles”.

A. Upload as Text File

Save each bundle contents in your text editor as a bundle.txt file, and upload it here into this discussion thread.

B. Insert as Formatted Code Block

If you are unable to upload files on this forum, alternatively you can paste the formatted bundles.
These are the two ways to format text so that it won’t be modified by the forum:

  • Method 1: Type code block manually

    Add three backticks ``` before and after the content/bundle, like this:

    ```
    content goes here
    ```

  • Method 2. Highlight and click the format button in the editor

Providing the input/output bundles will allow others to replicate what is going on in the scenario even if they do not use the external service.

Following these steps will allow others to assist you here. Thanks!

4

Here are output bundles, first module is RSS, than I retrieve HTML with HTTP module and then try to extract origin links with text-parser. But the problem is, that origin links are not in HTML and “follow redirected links” also does not work with Google news links.
3_text_parser_output.txt (5.5 KB)
2_http_module_output.txt (261.2 KB)
1_rss_module_output.txt (4.7 KB)

5

Does anyone have the idea how to get an original link from Google News page?

6

I’ve found a method, you have to use a hosted scraper like DumplingAI or ScrapeNinja.

Here is an example:

Screenshot_2024-08-30_150854

Example Output

Screenshot_2024-08-30_150844
Screenshot_2024-08-30_150844832×587 34.3 KB

Others

I’ve used ScrapeNinja, and you can use jQuery-like selectors in the extractor function. But this is more advanced. ScrapeNinja also can run the page in a real web-browser, loading all the content and running the page load scripts so it closely simulates what you see, as opposed to just the raw page HTML fetched from the HTTP module. If you want a ScrapeNinja example, take a look at Grab data from page and url - #5 by samliew

For more information on other different methods of web scraping, see Overview of Different Web Scraping Techniques in Make 🌐

Hope this helps! Let me know if there are any further questions or issues.

— @samliew

P.S.: Did you know, the concepts of about 70% of questions asked on this forum are already covered in the Make Academy. Investing some effort into it will save you lots of time and frustration using Make later!

7

super, thank you so much!!!