Trying to make several requests to the API and use the pagination offset but the same records are being returned

[news_bib]

What is your goal?

Make several requests to the API and insert the values in a sheet.

What is the problem & what have you tried?

The modules are a tool, a http client and a sheet. The offset on the Http client has been set eg 0, 100, 200,…,10,000. The scenario is set to run every 2 minutes with the aim of pulling new records. However, the same records are being added to the sheet. It’s the first 100 batch repeated. How do I fix this please? The tool module has no parameter (all errors are skipped on it). The http client has all the parameters.

How do I fix this?

Error messages or input/output bundles

No error message. The same records are inserted into the sheet which defeats the purpose and wastes credits.

Screenshots (scenario flow, module settings, errors)

[damato]

Hi @news_bib.
Welcome to the community!

Every time the scenario runs, the values set with Set Multiple Variables are reset. They are not retained between executions.

To query multiple pages, you can either:

1. Find out how many pages are there, in total, then iterate that many times, updating the page number on each iteration. This would be a single execution and would save you a lot of credits. You can iterate, make the http request, aggregate and the insert all the aggregated results in bulk.

2. If you want to fetch a different page on every execution, you need a way to save the state between scenario runs. Take a look at this post to achieve that:

How do I increment the value of a parameter in an http request each time the scenario is run starting with a certain value?

Hi @secretagency Whenever you set the variable inside the scenario itself, it will start with the hardcoded value every time. If you want it to be persistent between executions, you need to store it permanently somewhere else. You can use make’s own Data Store, creating a data structure that only holds a number. Or you can use anything else that can hold a value, like a Google spreadsheet. Just remember to read the value, increase it by 1 and update it on every run. @damato

@damato

[news_bib]

Thank you for your answer but I still don’t know how to do this? Maybe a diagram or something would help. I tried this article but it repeats records too Need help with sending req to API mutliple times

[Donald_Mitchell]

To further help with credits usage, you can use the Google Sheets - Bulk Add Rows (Advanced) which can add hundreds of rows in a single call/credit. There’s theoretically no limit other than the limit of cells you can have in a sheet, but a few hundred updates at a time is reasonable.

Do you intend to get all records from the API once then dump the results into Google Sheets and it never has to run again?

Or is this something you will want to run periodically to get new records from the API and add only the new records to Google Sheets?

[news_bib]

Thanks @Donald_Mitchell thank you for the Bulk Add I’ll try that. I would have liked to run it once and get all the records but there is an API limit of 100. So I have to keep making requests.

[Donald_Mitchell]

The Make an API call module now supports most forms of pagination, and in my experience with it, it only uses 1 credit to retrieve up to 3200 items. What you’re able to do exactly depends on the API you’re working with.

[news_bib]

@Donald_Mitchell I am working with an API on rapidapi with a limit of 100. So I need to make several call. It supports pagination in multiples of 100.

[news_bib]

@damato would you mind explaining further with a diagram or something?

[Donald_Mitchell]

@damato

To query multiple pages, you can either:

1. Find out how many pages are there, in total, then iterate that many times, updating the page number on each iteration. This would be a single execution and would save you a lot of credits. You can iterate, make the http request, aggregate and the insert all the aggregated results in bulk.
2. If you want to fetch a different page on every execution, you need a way to save the state between scenario runs. Take a look at this post to achieve that:

This is the old way of manually paginating, but HTTP now natively supports many forms of pagination.

[damato]

Thank you for the update information and for auditing my last 5 answers. Looks like you took something personally.
Sorry, I’m not doing this. I hate toxic environments.