Adding and testing pagination in custom app

Custom Apps
1

I’m building a Custom App and I see the app review prerequisites require configuring pagination. I’m working through this doc to configure my modules: Pagination | Make Apps

However I’m unsure how to actually use the pagination settings I’m adding to my app in a Scenario. The only pagination guide I could find is this: https://www.msquare.pro/post/api-pagination

But from what I can tell, this guide works regardless of whether the Module has embedded pagination or not.

Our APIs are a mix of pagination with next_page_tokens and num & start style. I’m starting with next_page_token style.

The Make docs say: In every following request, we’ll use the nextPageToken provided with the previous one. We’ll stop the pagination when no other nextPageToken was received.

This makes me think it’s a native feature I can enable in my Custom App and Make will automatically process the pagination until there are no tokens left. I can’t figure out how to create this directive though.

Here’s how my module is configured:

{
  "url": "/search.json",
  "method": "GET",
  "headers": {},
  "qs": {
    "engine": "google_jobs",
    "q": "{{parameters.q}}",
    "location": "{{parameters.location}}",
    "uule": "{{parameters.uule}}",
    "google_domain": "{{parameters.google_domain}}",
    "gl": "{{parameters.gl}}",
    "hl": "{{parameters.hl}}",
    // "next_page_token": "{{parameters.next_page_token}}",
    "chips": "{{parameters.chips}}",
    "lrad": "{{parameters.lrad}}",
    "ltype": "{{parameters.ltype}}",
    "no_cache": "{{parameters.no_cache}}",
    "async": "{{parameters.async}}",
    "output": "{{parameters.output}}"
  },
  "pagination": {
    "qs": {
  	  "next_page_token": "{{body.serpapi_pagination.next_page_token}}"
    },
    "condition": "{{body.serpapi_pagination.next_page_token}}"
  }
}

Any advice?

2

Hello,

can you show us an example query and response where we can see how nextpagetoken is configured and returned?

Benjamin

1 Like
3

Thank you for the response!

The initial request would be a GET to:

https://serpapi.com/search.json?engine=google_jobs&q=Barista&google_domain=google.com&api_key=123...

The response body to the above request would include a next_page_token.

{
        "search_metadata":
        { … }
        ,
        "search_parameters":
        { … }
        ,
        "jobs_results":
        [ … ]
        ,
        "serpapi_pagination":
        {
            "next_page_token": "eyJmYyI6IkVzd0VDb3dFUVVwSE9VcHJUeTE1TUVoU2RtbFJOa2hQZG0xNVdHZEJVRFV5UkhZMFpsSnFNMU5XUjNZNGNsZ3lWa1JGU2kwd2VuTnhjVGxHZFROeExUVklMWGxFY0VVeWN6VkllR1JXYkVoSFJVaE1OMnRNYTJsdVowcDVlWHBHVERKMk0yRnVZVzVMVWpCNGRUZHBYMEpsVWxCVFRsRkNWek42WVVJM1VYQmhNRFZYTVZsWWMyNTVNMUp0WmkxT1MzSXdjR2xpVW1kSlpGWTNZbTV1YW0xZlNEVXpTMVl4TTIxbmJISkdZMFJTV2tzeVdXdE1aa281VURSWGVrUjRSM05pVUdkbVQwMTFkVUpsUVZKRVFWUnhiVTVGU2xORFIwdDFSVFJzWjNkM1ZGZDRVbWxoUnpOalVrWnFlbEE0Vm5aMWNVWmZMVFU1Tms5UVozcFRSMlZsYUZONVFrTklMVzV6ZDBSek1XTndTME42VjFFNGFubHJkMUpmVEhJemR6bDFPVTkxWkZGV09VaEZSbkZ0TFhZeVpraDJTMnAzTVRjeGFHMDFVMGN5YUZkTVkzSnFZVXRSVDNWM2FFOUVRak5XU1c5SVdUTmFjVlJWZVRsM2FYbHVaVWhJT0Y5V2JsTnJZMngwZURRM2NqUnFTRXRoUlVweU0zSnVjRWMzU1dJeVdtbzBSbVJCUTFoTUxYZzVVbkJyTUdaWExWYzNRM0pNZVhSYVZFWmtMV1pQWXpnek5ETjJNVEowVG1WdE9XRnpVVE5xVUZrMloyYzRka0l0U0VSaWEyWjZTR2xQU2pObk0xSkVORTAyTlVZeGFHTk5RM2N4Tm0xS1EwdDFYMDR3U1U4eVZGWlZOV2h6U1dscVNGUmFaM2xOVURSQmRYUkZRVE5PT1MxWFR6ZzFYMlJCUjFsM2FUSTJiazk0TkdrU0YyTktTRVZhZG1wMFNFcEhaelZPYjFCdWNUSlVhVUZGR2lKQlJsaHlSV052VTBocWRFUkphRFZLWVZaSGMySXdVSEJKWmtNMU9WUkxUazEzIiwiZmN2IjoiMyJ9",
            "next": "https://serpapi.com/search.json?engine=google_jobs&google_domain=google.com&next_page_token=eyJmYyI6IkVzd0VDb3dFUVVwSE9VcHJUeTE1TUVoU2RtbFJOa2hQZG0xNVdHZEJVRFV5UkhZMFpsSnFNMU5XUjNZNGNsZ3lWa1JGU2kwd2VuTnhjVGxHZFROeExUVklMWGxFY0VVeWN6VkllR1JXYkVoSFJVaE1OMnRNYTJsdVowcDVlWHBHVERKMk0yRnVZVzVMVWpCNGRUZHBYMEpsVWxCVFRsRkNWek42WVVJM1VYQmhNRFZYTVZsWWMyNTVNMUp0WmkxT1MzSXdjR2xpVW1kSlpGWTNZbTV1YW0xZlNEVXpTMVl4TTIxbmJISkdZMFJTV2tzeVdXdE1aa281VURSWGVrUjRSM05pVUdkbVQwMTFkVUpsUVZKRVFWUnhiVTVGU2xORFIwdDFSVFJzWjNkM1ZGZDRVbWxoUnpOalVrWnFlbEE0Vm5aMWNVWmZMVFU1Tms5UVozcFRSMlZsYUZONVFrTklMVzV6ZDBSek1XTndTME42VjFFNGFubHJkMUpmVEhJemR6bDFPVTkxWkZGV09VaEZSbkZ0TFhZeVpraDJTMnAzTVRjeGFHMDFVMGN5YUZkTVkzSnFZVXRSVDNWM2FFOUVRak5XU1c5SVdUTmFjVlJWZVRsM2FYbHVaVWhJT0Y5V2JsTnJZMngwZURRM2NqUnFTRXRoUlVweU0zSnVjRWMzU1dJeVdtbzBSbVJCUTFoTUxYZzVVbkJyTUdaWExWYzNRM0pNZVhSYVZFWmtMV1pQWXpnek5ETjJNVEowVG1WdE9XRnpVVE5xVUZrMloyYzRka0l0U0VSaWEyWjZTR2xQU2pObk0xSkVORTAyTlVZeGFHTk5RM2N4Tm0xS1EwdDFYMDR3U1U4eVZGWlZOV2h6U1dscVNGUmFaM2xOVURSQmRYUkZRVE5PT1MxWFR6ZzFYMlJCUjFsM2FUSTJiazk0TkdrU0YyTktTRVZhZG1wMFNFcEhaelZPYjFCdWNUSlVhVUZGR2lKQlJsaHlSV052VTBocWRFUkphRFZLWVZaSGMySXdVSEJKWmtNMU9WUkxUazEzIiwiZmN2IjoiMyJ9&q=Barista"
        }
    }

You then extract the next_page_token and append it as a param to the original request like so:

https://serpapi.com/search.json?engine=google_jobs&q=Barista&google_domain=google.com&next_page_token=eyJmYyI6IkVzd0VDb3dFUVVwSE9VcHJUeTE1TUVoU2RtbFJOa2hQZG0xNVdHZEJVRFV5UkhZMFpsSnFNMU5XUjNZNGNsZ3lWa1JGU2kwd2VuTnhjVGxHZFROeExUVklMWGxFY0VVeWN6VkllR1JXYkVoSFJVaE1OMnRNYTJsdVowcDVlWHBHVERKMk0yRnVZVzVMVWpCNGRUZHBYMEpsVWxCVFRsRkNWek42WVVJM1VYQmhNRFZYTVZsWWMyNTVNMUp0WmkxT1MzSXdjR2xpVW1kSlpGWTNZbTV1YW0xZlNEVXpTMVl4TTIxbmJISkdZMFJTV2tzeVdXdE1aa281VURSWGVrUjRSM05pVUdkbVQwMTFkVUpsUVZKRVFWUnhiVTVGU2xORFIwdDFSVFJzWjNkM1ZGZDRVbWxoUnpOalVrWnFlbEE0Vm5aMWNVWmZMVFU1Tms5UVozcFRSMlZsYUZONVFrTklMVzV6ZDBSek1XTndTME42VjFFNGFubHJkMUpmVEhJemR6bDFPVTkxWkZGV09VaEZSbkZ0TFhZeVpraDJTMnAzTVRjeGFHMDFVMGN5YUZkTVkzSnFZVXRSVDNWM2FFOUVRak5XU1c5SVdUTmFjVlJWZVRsM2FYbHVaVWhJT0Y5V2JsTnJZMngwZURRM2NqUnFTRXRoUlVweU0zSnVjRWMzU1dJeVdtbzBSbVJCUTFoTUxYZzVVbkJyTUdaWExWYzNRM0pNZVhSYVZFWmtMV1pQWXpnek5ETjJNVEowVG1WdE9XRnpVVE5xVUZrMloyYzRka0l0U0VSaWEyWjZTR2xQU2pObk0xSkVORTAyTlVZeGFHTk5RM2N4Tm0xS1EwdDFYMDR3U1U4eVZGWlZOV2h6U1dscVNGUmFaM2xOVURSQmRYUkZRVE5PT1MxWFR6ZzFYMlJCUjFsM2FUSTJiazk0TkdrU0YyTktTRVZhZG1wMFNFcEhaelZPYjFCdWNUSlVhVUZGR2lKQlJsaHlSV052VTBocWRFUkphRFZLWVZaSGMySXdVSEJKWmtNMU9WUkxUazEzIiwiZmN2IjoiMyJ9&next_page_token=eyJmYyI6IkVzd0VDb3dFUVVwSE9VcHJUeTE1TUVoU2RtbFJOa2hQZG0xNVdHZEJVRFV5UkhZMFpsSnFNMU5XUjNZNGNsZ3lWa1JGU2kwd2VuTnhjVGxHZFROeExUVklMWGxFY0VVeWN6VkllR1JXYkVoSFJVaE1OMnRNYTJsdVowcDVlWHBHVERKMk0yRnVZVzVMVWpCNGRUZHBYMEpsVWxCVFRsRkNWek42WVVJM1VYQmhNRFZYTVZsWWMyNTVNMUp0WmkxT1MzSXdjR2xpVW1kSlpGWTNZbTV1YW0xZlNEVXpTMVl4TTIxbmJISkdZMFJTV2tzeVdXdE1aa281VURSWGVrUjRSM05pVUdkbVQwMTFkVUpsUVZKRVFWUnhiVTVGU2xORFIwdDFSVFJzWjNkM1ZGZDRVbWxoUnpOalVrWnFlbEE0Vm5aMWNVWmZMVFU1Tms5UVozcFRSMlZsYUZONVFrTklMVzV6ZDBSek1XTndTME42VjFFNGFubHJkMUpmVEhJemR6bDFPVTkxWkZGV09VaEZSbkZ0TFhZeVpraDJTMnAzTVRjeGFHMDFVMGN5YUZkTVkzSnFZVXRSVDNWM2FFOUVRak5XU1c5SVdUTmFjVlJWZVRsM2FYbHVaVWhJT0Y5V2JsTnJZMngwZURRM2NqUnFTRXRoUlVweU0zSnVjRWMzU1dJeVdtbzBSbVJCUTFoTUxYZzVVbkJyTUdaWExWYzNRM0pNZVhSYVZFWmtMV1pQWXpnek5ETjJNVEowVG1WdE9XRnpVVE5xVUZrMloyYzRka0l0U0VSaWEyWjZTR2xQU2pObk0xSkVORTAyTlVZeGFHTk5RM2N4Tm0xS1EwdDFYMDR3U1U4eVZGWlZOV2h6U1dscVNGUmFaM2xOVURSQmRYUkZRVE5PT1MxWFR6ZzFYMlJCUjFsM2FUSTJiazk0TkdrU0YyTktTRVZhZG1wMFNFcEhaelZPYjFCdWNUSlVhVUZGR2lKQlJsaHlSV052VTBocWRFUkphRFZLWVZaSGMySXdVSEJKWmtNMU9WUkxUazEzIiwiZmN2IjoiMyJ9&api_key=123...
4

Thanks!

At first, you seem to have done correctly. What is the behavior of the module? Does it break? returns you only 1 page?

One way to debug it is to use the Make Dev tool (Make dev tool) and to debug the calls the module is doing. It will help you see whether the next_page_token is here or not in the response and wheter it’s picked by the module or not.

If you never used it, here is an example of what the dev tool looks like

image
image2314×1228 326 KB

Benjamin

1 Like
5

The current behavior is the Module only makes 1 call which gets the initial page. There doesn’t appear to be any automated attempt to get the next page.

And I’m not sure how to direct my Scenario to keep getting pages outside of that tutorial I posted earlier which is a separate thing from what I can tell.

If I add a duplicate of the Module after the first instance, it just runs the same initial call to grab the first page.

Is there a way to direct Make to use the pagination configured in the Module settings?

I haven’t tried the Make Dev Tool yet since I don’t have Chrome installed at the moment. At this point, I don’t think there’s much to debug there anyway because my Scenario is only grabbing the first page and not trying anything else. I can try that if you think it’s necessary to track down whether a second call is being made for pagination.

Thank you again!

6

Hello,

The way you configured the module should have Make grab automatically the next pages.
The DevTool can be interesting to see with more details what the module sends for the request and what it gets back from the API.

I suspect there is something wrong with the condition.

Let me DM you.

Benjamin

7

Thanks Benjamin! Just replied to your DM.

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.