Assistance with parsing URLs and iterating on each one

[Sean_Walker]

I have a scenario where PerplexityAI [30] searches the web to find links with information relevant to upstream information. It passes on a list of urls:

[
{
“id”: “0230c8e0-912d-48f0-b96e-35e1f049a7f4”,
“model”: “llama-3-sonar-large-32k-online”,
“created”: 1725555481,
“usage”: {
“prompt_tokens”: 255,
“completion_tokens”: 393,
“total_tokens”: 648
},
“object”: “chat.completion”,
“choices”: [
{
“index”: 0,
“finish_reason”: “stop”,
“message”: {
“role”: “assistant”,
“content”: “1. How does architectural visualization differ fundamentally from traditional architectural renderings?\n - https://fortes.vision/blog/the-role-and-advantages-of-3d-architectural-visualization-for-architects/\n\n2. What are the most common techniques used in Photoshop for architectural visualization?\n - https://www.photoshoptrainingchannel.com/architectural-visualization-photoshop-tutorial/\n\n3. How do advances in software tools like LUMION and Enscape impact the architectural visualization industry?\n - Industry-Leading 3D Rendering Software | Lumion - Lumion 3D Rendering Software. What are the current limitations of VR technology in the field of architectural visualization?\n - https://www.archdaily.com/932242/the-limits-of-virtual-reality-in-architecture\n\n5. How do architectural visualization studios structure their workflow for complex projects?\n - https://www.thespaces.com/2022/04/28/architectural-visualization-workflow/\n\n6. What are the primary skills required for someone looking to specialize in architectural visualization?\n - https://www.cgarchitect.com/features/skills-required-for-architectural-visualization\n\n7. What role does narrative and emotional engagement play in architectural visualization?\n - https://www.dezeen.com/2022/07/21/architectural-visualization-narrative-emotional-engagement/\n\n8. How does client feedback typically influence the final visualization output in architectural projects?\n - https://www.architecturaldigest.com/story/client-feedback-architectural-visualization\n\n9. What are the potential future trends in architectural visualization technology that might disrupt the industry?\n - RMJM Creates a 180 Meters Tower for the Xiangjiang Gate in Hunan, China | ArchDaily”
},
“delta”: {
“role”: “assistant”,
“content”: “”
}
}
]
}
]

This goes into a ChatGPT module [42] which cleans up the output and returns each of the urls in the format of ‘url1="this-is-url1.com’ and ‘url2=this-is-the-next-link.com’

You get the idea.

{
    "result": "url1=\"https://fortes.vision/blog/the-role-and-advantages-of-3d-architectural-visualization-for-architects/\"\nurl2=\"https://www.photoshoptrainingchannel.com/architectural-visualization-photoshop-tutorial/\"\nurl3=\"https://www.lumion.com/blog/10-ways-lumion-11-will-revolutionize-your-architectural-visualizations/\"\nurl4=\"https://www.archdaily.com/932242/the-limits-of-virtual-reality-in-architecture\"\nurl5=\"https://www.thespaces.com/2022/04/28/architectural-visualization-workflow/\"\nurl6=\"https://www.cgarchitect.com/features/skills-required-for-architectural-visualization\"\nurl7=\"https://www.dezeen.com/2022/07/21/architectural-visualization-narrative-emotional-engagement/\"\nurl8=\"https://www.architecturaldigest.com/story/client-feedback-architectural-visualization\"\nurl9=\"https://www.archdaily.com/931111/the-future-of-architectural-visualization-trends-and-predictions\"",
    "id": "chatcmpl-A4A5xSp7DysgaZAVbUpSLVfVLoALP",
    "object": "chat.completion",
    "created": "2024-09-05T16:58:01.000Z",
    "model": "gpt-4o-mini-2024-07-18",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "url1=\"https://fortes.vision/blog/the-role-and-advantages-of-3d-architectural-visualization-for-architects/\"\nurl2=\"https://www.photoshoptrainingchannel.com/architectural-visualization-photoshop-tutorial/\"\nurl3=\"https://www.lumion.com/blog/10-ways-lumion-11-will-revolutionize-your-architectural-visualizations/\"\nurl4=\"https://www.archdaily.com/932242/the-limits-of-virtual-reality-in-architecture\"\nurl5=\"https://www.thespaces.com/2022/04/28/architectural-visualization-workflow/\"\nurl6=\"https://www.cgarchitect.com/features/skills-required-for-architectural-visualization\"\nurl7=\"https://www.dezeen.com/2022/07/21/architectural-visualization-narrative-emotional-engagement/\"\nurl8=\"https://www.architecturaldigest.com/story/client-feedback-architectural-visualization\"\nurl9=\"https://www.archdaily.com/931111/the-future-of-architectural-visualization-trends-and-predictions\"",
                "refusal": null
            },
            "logprobs": null,
            "finish_reason": "stop"
        }
    ],
    "usage": {
        "prompt_tokens": 544,
        "completion_tokens": 238,
        "total_tokens": 782
    },
    "system_fingerprint": "fp_f33667828e"
}

]

I want to send each of these urls one at a time to another instance of PerplexityAI [38] and have it search and provide a summary of what it finds at each of the urls using Google docs [31]

I have watched multiple videos and searched the forums and I can’t figure this out. I know that I can use an iterator, but I can’t figure how to set it up. I’ve used text parsers, I’ve used arrays, I’ve tried splitting the text, but I can’t come up with the right combination of rules and variables to make it all work. Any assistance would be greatly appreciated. I’ve attached the JSON for the entire scenario.

blueprint (2).json (59.2 KB)

[samliew]

@Sean_Walker

This goes into a ChatGPT module [42] which cleans up the output

You don’t need this.

Please re-upload the first code block formatted properly so I can create an example for you.

1. This forum might have or already changed your text

When pasting text into this forum, you should format the example text using the rich-text editor, otherwise the forum software might modify the displayed text, and you might get incorrect answers from others because of it.

Some things this forum software might do to mangle your text:

– remove extra spaces (which may be necessary)
– convert links to titles (when copied is incorrect)
– incorrect joined links
– convert single and double quotes to smart angled quotes (“ ”)
– emojis
– etc.

This interferes with you receiving correct answers, because it:

– makes JSON invalid (you can verify when copy-paste into https://jsonformatter.org)
– makes incorrect text examples when we need to build a pattern for text parsing

2. To prevent this in future, please format text in code blocks

These are the two ways to format text so that it won’t be modified by the forum:

• Method 1: Type code block manually

  Add three backticks ``` before and after the content/bundle, like this:

  ```
  <sup>content goes here</sup>
  ```

• Method 2. Highlight and click the format button in the editor

  

3. You might need to re-copy the original text

Once the post has been submitted, it’s too late to format it since it’s already butchered, and you need to make a re-copy of the text, and format it before submitting the forum post.

Please let us know once you have corrected the issue. This will avoid others potentially providing wrong answers based on incorrect text in your question.

Thank you!

Hope this helps! Let me know if there are any further questions or issues.

— @samliew

P.S.: Investing some effort into the Make Academy will save you lots of time and frustration using Make.

[samliew]

Never mind, you can just use the Text Parser “Match Elements” module to get the URLs from the first output. You do not need to “clean” anything.

Hope this helps! Let me know if there are any further questions or issues.

— @samliew

P.S.: Investing some effort into the Make Academy will save you lots of time and frustration using Make.

[Sean_Walker]

Here is the current iteration of the scenario:

blueprint (4).json (71.7 KB)

I have to upload it BC it is too long and the forum won’t let my post be that long.

Here is what I am trying to accomplish:

Perplexity has taken a list of questions and searched to find content that answers the questions. It returns a message with a typical AI “Here are the answers to your questions” response:

[
    {
        "id": "b9e70847-5e6b-41b4-a15e-d5b0f1cbb4c2",
        "model": "llama-3-sonar-large-32k-online",
        "created": 1725987213,
        "usage": {
            "prompt_tokens": 305,
            "completion_tokens": 469,
            "total_tokens": 774
        },
        "object": "chat.completion",
        "choices": [
            {
                "index": 0,
                "finish_reason": "stop",
                "message": {
                    "role": "assistant",
                    "content": "Here are the search results for the given questions:\n\n1. **Benefits of incorporating natural elements in office spaces**:\n   - https://hbr.org/2023/07/research-a-little-nature-in-the-office-boosts-morale-and-productivity\n   - https://www.flipspaces.com/us/blogs-details/natural-elements-enhance-employees-productivity-in-office-space\n   - https://anthamgroup.com/best-ways-to-include-natural-elements-in-your-office-design/\n   - https://blog.turningart.com/how-to-use-natural-elements-to-create-a-productive-office-space\n   - https://www.haiken.com/insights/organic-office-spaces-enhancing-wellbeing-with-natural-elements\n\n2. **Impact of office color on collaboration and communication**:\n   - https://www.entrepreneur.com/article/292644\n   - https://www.fastcompany.com/3044449/the-psychology-of-color-in-the-workplace\n\n3. **Color schemes for creative industries vs. analytical work environments**:\n   - https://www.99designs.com/blog/color-psychology/color-meaning/\n   - https://www.canva.com/learn/color-theory-in-design/\n\n4. **Practical tips for selecting colors for a small office space**:\n   - https://www.thespruce.com/office-color-schemes-4127189\n   - https://www.hunker.com/13428444/how-to-choose-colors-for-a-small-office\n\n5. **Balancing personal favorite colors with scientifically proven productivity colors**:\n   - https://www.colorpsychology.org/color-and-productivity/\n   - https://www.verywellmind.com/color-psychology-2795028\n\n6. **Latest trends in home office design influenced by remote work**:\n   - https://www.architecturaldigest.com/story/home-office-design-trends\n   - https://www.bloomberg.com/news/articles/2022-02-24/remote-work-is-changing-home-office-design\n\n7. **Aligning office interior colors with company branding and logo colors**:\n   - https://www.99designs.com/blog/branding/brand-color-palette/\n   - https://www.canva.com/learn/brand-colors/"
                },
                "delta": {
                    "role": "assistant",
                    "content": ""
                }
            }
        ]
    }
]

I want to save that response in a document that I can refer to down the road and in other scenarios, so I have two Google Docs modules [31] and [52] after that.

Then I want to get each individual URL in it’s own bundle so that I can send each URL to ChatGPT and have it go to that URL, summarize the content at the link and perform other actions downline based on the content found at that link.

So, I have a Chat GPT module [42] that takes the typical AI response from Perplexity [30] and only gives me the URLs. The original reason I’m doing this is because the Text Parser [60] wasn’t doing what I wanted and I was trying to get the URLs formatted in a way that I could use a regular expression to cleanly parse them out. It may not be needed as mentioned above.

Then I have a Text Parser [60] that is supposed to take the output from Chat GPT [42] and separate each link into it’s own bundle, pass it to Repeater [61] and finally Chat GPT [62] has a prompt (it’s a stupid prompt right now, ignore it) that tells it to look at the link, go to that site and summarize the information there. Eventually, I’ll store that information in a google doc also.

The problem that I’m running into and cannot figure out is this…

Sometimes the Text Parser [60] is correctly like this:

THE INPUT TO IT

[
    {
        "text": "url1=\"https://www.benjaminmoore.com/en-us/project-ideas-inspiration/interiors/home-office-paint-colors\"\nurl2=\"https://www.trucolorscontracting.com/boost-productivity-with-14-professional-home-office-paint-colors\"\nurl3=\"https://www.oahupropainters.com/blog/10-best-home-office-paint-colors-to-boost-productivity\"\nurl4=\"https://cobaltworkspace.com/best-colors-for-focus-and-productivity/\"",
        "pattern": "##http_urls",
        "requireProtocol": true,
        "specialCharsPattern": null
    }
]

THE OUTPUT OF IT

[
    {
        "match": "https://www.benjaminmoore.com/en-us/project-ideas-inspiration/interiors/home-office-paint-colors"
    },
    {
        "match": "https://www.trucolorscontracting.com/boost-productivity-with-14-professional-home-office-paint-colors"
    },
    {
        "match": "https://www.oahupropainters.com/blog/10-best-home-office-paint-colors-to-boost-productivity"
    },
    {
        "match": "https://cobaltworkspace.com/best-colors-for-focus-and-productivity/"
    }
]

But the repeater is not set up right. It’s not getting the bundle from the Text Parser:

THE INPUT GOING INTO THE REPEATER:

If I can get the repeater to work that would be amazing! Thank you for your help and advice.

[samliew]

I think you meant to use an Iterator instead of a Repeater.

Also, you do not need an Iterator module, because each match from the Text Parser is already in its own bundle.

You could probably just delete the Repeater module.

Hope this helps! Let me know if there are any further questions or issues.

— @samliew

P.S.: Investing some effort into the Make Academy will save you lots of time and frustration using Make.

[Sean_Walker]

Ok.

I have removed the repeater all together and you are absolutely correct. The text parser is putting each url into it’s own bundle and passing each one onto the next module, one at a time. I thought that I needed to use a repeater or iterator and couldn’t figure out how to do it.