Having issues in apllying pagination for the flow

[Afzal]

Goal:

I’m currently facing challenges in applying pagination within my workflow. I aim to implement pagination to fetch data from a source efficiently while preventing infinite loops and handling empty responses appropriately.

Steps taken:

1. Set up a variable to store pagination information.
2. Configured an HTTP module to fetch data from the source.
3. Applied the pagination variable in the query parameter of the HTTP module to retrieve specific pages of data.
4. Implemented safeguards to prevent infinite loops by setting conditions to stop the pagination process.
5. Created filters to handle empty responses gracefully.
6. Utilized an HTTP module with filtering capabilities to check for empty data.

Screenshots:

Issues:

Despite the setup and configurations implemented, I’m encountering difficulties in effectively applying pagination within the workflow. The workflow may be entering infinite loops or not handling empty responses as intended. Any insights or suggestions to resolve these issues would be appreciated.

[samliew]

To handle (ignore) the warning if the HTTP module fails to fetch data, simply add an Error Handler to it and attach an “Ignore” directive module.

By handling any errors by adding a “Error handler” to the module, the scenario won’t throw an error and get turned off.

For more information on how to handle errors, see:

• Introduction to Error Handling
• Directives for Error Handling
• Advanced Error Handling
• Watch these videos by Make - Learn all about error handling (YouTube playlist)

[hideyukishibata]

Hmm your scenario doesn’t look like it’s configured like it’s supposed to. If you could leave the blueprint here, I’d be happy to take a look inside.

[Stoyan_Vatov]

Hi Afzal,

what type of pagination are you facing?

Is it an incremental number of pages or links to next page or something different?

We usually handle pagination using datastores to keep track of the page we reached outside of the scenario it self, maybe you can look into creating something similar?

Basically the scenario overwrites the next entry value in the data store and then it keeps running until it reaches the last page.

[Afzal]

Hi @samliew
Scene is , I have configured the pagination using below link

Master Pagination Techniques of API in Make (Formerly Integromat)

And by following the steps, when I run for the pagination, it’s working, but when for the first time scenario is running, it’s taking wrong pagination steps ( as shown below afterId )

And for error handling I have setup a filter to lookup data.message ( it comes when there is no data from api itself, either it comes with 4XX or 5XX status code ) exist or not.

So problem is causing when first time page value is being set which is not correct and applying with filter on error handler , scenario terminates in the first loop.

[Afzal]

Hi @Stoyan_Vatov
It’s issue with the incremental number of pages, not the links to the next page or something.

We need to set the initial value for the page to 1 and it should be increment by 1 like 2,3,4,5,6,7,8,9…,upto last page when data is getting empty from the api response and when we get empty response from the api, it should break the scenario loop and and end that.

But while applying the filter for the error handler loop, it’s ending in the first loop itself, because for the first time page is undefined and from api response it’s returning with status code 4XX .

[hideyukishibata]

Hey @Afzal my apologies for the late response, but here it is. I had a look at the blueprint you DMed me and fixed it for you. The root problem you had in it was this (HTTP > Make a request):

If you have this on, you can make requests but query strings don’t work and whatever parameters you throw in, you simply get all the quotes.

You might want to do this instead:

You might have had a problem around Repeater as well but not too sure as I had started fixing already and did not get to see it. Anyway, it’s all fixed, simplified, and made versatile.

DummyJSON pagination.json (14.4 KB)