Filtering HTTP response

[statipletus]

Hello, I’m scraping my own web database with “Make A Request” Module.
I managed to get my data right.

However, I don’t know one silly thing.

That is, what filter to choose when I want my scenario to start if only existing data is not scraped already. Because my scenario is keep repeating itself over and over. I want to filter it right at the start that my operations wouldn’t be consumed.

I tried to include a filter like this: “If Incident date does not exist” (because I’m scraping incidents) and “Data id does not exist”. However both were wrong, because either way automation doesn’t start with a new record as well.

What should I do?

My complete scenario look like this:

[NP]

Hi, can you explain a bit more about the data you are filtering? When new records are added to the database, are they added with the incident date or data ID already in place? If so, that could explain why new records also aren’t being processed.

[statipletus]

They are added manually, when a new event happened. All new events has unuque data ID.
I don’t know why new records aren’t processed.

Here’s how my filter look like when I added a new record, which was not processed already:

That’s the data I’m scraping (it’s just a bit of it):

I’m pretty new to make. Do I need to provide additonal info?

[NP]

The red emoji means that the bundle did not pass through the filter (so either the incident date or data ID already exists (or both already exist) and the scenario is filtering it out before the next step is reached). I would take another look at the specific data fields going into your scenario, it’s like you will find the answer once you double-check that!

[statipletus]

Thank you. I’m wrong somewhere, but I don’t know where.

I have info about data EVENT ID and INCIDENT DATE that is always unique in my database. So basically, all this info should be filtered according to unique data standart. And if this data meets this standart, it should be proceeded only. I don’t want that scenario would be going if HTTP module already scanned the same EVENT ID column.

In excel sheet it’s very easy to make the same filter, for example, I could make filter according to “Status” or “Event ID” columns (if they exist).

My problem is that I don’t know how to apply the same logic to the first step of the workflow - HTTP module.

How would you structure your filter if you’d like to be sure that HTTP module scans only new records, which it didn’t scan in the past?

[NP]

So even though the event ID and incident date are unique, if they already exist when the record is passed and your filter is only for records where these fields are blank, no records will pass regardless of whether the values are unique or not.

I think it would probably be easier to just send a webhook that starts your scenario when new records are added.

[samliew]

Looks like you have to move the filter to after the iterator module instead, so that you are filtering each item in the array correctly.

Currently you are only checking if the first item is valid, if yes, process all items in the array.

If you need further assistance, please provide the following:

1. Screenshots of HTTP module fields

Please share screenshots of the HTTP trigger module fields in question? It would really help other community members to see what you’re looking at.

You can upload images here using the Upload icon in the text editor:

2. Scenario blueprint

Please export the scenario blueprint file to allow others to view the mappings and settings. At the bottom of the scenario editor, you can click on the three dots to find the Export Blueprint menu item.

^{(Note: Exporting your scenario will not include private information or keys to your connections)}

Uploading it here will look like this:

blueprint.json (12.3 KB)

3. And most importantly, output bundles

Please provide the output bundles of the HTTP trigger module by running the scenario (or get from the scenario History tab), then click the white speech bubble on the top-right of each module and select “Download input/output bundles”.

A.

Save each bundle contents in your text editor as a bundle.txt file, and upload it here into this discussion thread.

Uploading them here will look like this:

module-1-input-bundle.txt (12.3 KB)
module-1-output-bundle.txt (12.3 KB)

B.

If you are unable to upload files on this forum, alternatively you can paste the formatted bundles in this manner:

• Either add three backticks ``` before and after the code, like this:

  ```
  <sup>input/output bundle content goes here</sup>
  ```

• Or use the format code button in the editor:
  

Providing the input/output bundles will allow others to replicate what is going on in the scenario even if they do not use the external service.

Following these steps will allow others to assist you here. Thanks!

[statipletus]

Hello, I’m still having problem and filter after iterator module doesn’t solve my problem (or I don’t know how to use it properly).

I send you all the files you asked. If you’ll have any additional requirements, please say so.

P.S. I had to depersonalize data from some fields.
Model-1-output-bundle.json (15.1 KB)
Module-1-input-bundle.json (1.6 KB)
HTTP Filter Problem.json (12.3 KB)

[statipletus]

Sam, maybe you had some time to figure out possible solution for this specific use case?

[statipletus]

Hello NP,

How / where could I transform my HTTP module into Webhook? I don’t know how to do it. Do you know some resources? Below I added my files, maybe it will help you.