HTTP Pagination Loop

Questions
, ,
1

:bullseye: What is your goal?

There’s an API that I need to query and paginate (when there are many items).

The response I get from the API when there are more pages looks like this:

{
    "results": [list of items], 
    "paginationKey": {object that I need to add as is to the next request}
}

So in short what I need to do is:

  1. Send HTTP request.
  2. If no paginationKey in response - stop.
  3. If paginationKey exists - send another request with current “paginationKey”, and continue to aggregate the results.

Any ideas?

:thinking: What is the problem & what have you tried?

I tried using the new HTTP module’s pagination options, but they don’t fit this specific case.

I tried different scenarios with Repeater module, but the problem is that Make doesn’t allow me to “rewrite” the same variable (I tried setting a paginationKey variable and updating it each loop, but I can’t update the original variable).

3

For HTTP pagination loops, the standard approach is Repeater → HTTP → Router: one route continues if a next_page token exists, the other breaks the loop. Set the Repeater to a high max count, and in the break route use a filter that checks when the pagination token is empty. What type of pagination does your API use - offset, cursor, or page number? Check 4 API Pagination Patterns for Make.com — Page | Offset | Cursor | Link Header for patterns.

4

That would be the case a couple of months ago before the new HTTP module was released. Now the standard approach is to use the built in pagination in the HTTP module.

5

good point, I may have jumped to the Repeater approach a bit too quickly. Does the built-in pagination handle cases where the cursor is a whole object rather than a simple string token? that’s what Crumbs seems to be dealing with and I wasn’t sure if the new module covers that.

6

It should yeah. You can specify the item path where the pagination object is when you select Token or cursor-based Pagination type.

7

Yeah I tried that, but it didn’t work with a whole object. I guess it expects a specific value like string etc. I’ll check if I still have my scenario attempt, as since I just solved it by calling a backend server function for this.

8

interesting. so the built-in pagination does fall short with object cursors. the Repeater approach would actually handle that case, though I get why calling a backend function is cleaner if you have that option.

9

You should be able to map the paginationKey object in there and it should be able to call it to get to the next page. Can you share some screenshots of the setup and what the api returns as paginationKey?

10

I tried but when I couldn’t get it to work I just implemented some python backend code to do the pagination for me and return the results.

Maybe one day I’ll try it again through Make :wink:

Anyway here’s a screenshot of how the response pagination looks like:

image
image595×290 10.6 KB