How to divide up an http GET request into batches based on how many bundles are available

How To
1

The title is probably a bit dense, sorry.

I have a simple scenario which is:

HTTP GET (from my database)
→
Add New Row in Google Sheet for each bundle.

I have this working well with the Iterator function.

However the problem is that currently I have to put a limit on how many bundles the GET command retrieves, as there’s so many in the database that Make returns an error that the response exceeds the maximum file size.

I figure I can put a Repeater Function and some sort of maths function or increment or something to essentially do the following:

GET total record number
→
Divide record number into four or similar
→
GET records 1-[a quarter of total]
→ ADD to Google Sheets
→
GET records [a quarter of total+1]-[two quarters of total]
→ add to google sheets
→
GET records [two quarters +1]-[three quarters of total]

and so on.

Can anyone help me achieve this?

2

There is now a Bulk Add Rows module in Make for Google Sheets.

3

My issue is that the initial HTTP GET request is too large for Make to retrieve.

4

Welcome to the Make community!

You have asked a duplicate question of

If you need further assistance, please provide the following:

1. Relevant Screenshots

Please share screenshots of your scenario, any error messages, relevant module fields, and filters in question? It would really help other community members to see what you’re looking at.

You can upload images here using the Upload icon in the text editor:

2. Scenario Blueprint

Please export the scenario blueprint file to allow others to view the mapped variables in the module fields. At the bottom of the scenario editor, you can click on the three dots to find the Export Blueprint menu item.

3. Output Bundles of Modules

Please provide the output bundles of the modules by running the scenario (or get from the scenario History tab), then click the white speech bubble on the top-right of each module and select “Download input/output bundles”.

A. Upload as Text File

Save each bundle contents in your text editor as a bundle.txt file, and upload it here into this discussion thread.

B. Insert as Formatted Code Block

If you are unable to upload files on this forum, alternatively you can paste the formatted bundles.
These are the two ways to format text so that it won’t be modified by the forum:

  • Method 1: Type code block manually

    Add three backticks ``` before and after the content/bundle, like this:

    ```
    content goes here
    ```

  • Method 2. Highlight and click the format button in the editor

Providing the input/output bundles will allow others to replicate what is going on in the scenario even if they do not use the external service.

Following these steps will allow others to assist you here. Thanks!

Hope this helps! Let me know if there are any further questions or issues.

— @samliew

P.S.: Did you know, the concepts of about 70% of questions asked on this forum are already covered in the Make Academy. Investing some effort into it will save you lots of time and frustration using Make later!

5

There is a really good learning module on Make academy about pagination. Take a look at that and maybe it can help to setup a solution.

However are you trying to enter in old data into the Google Sheet? Or is this new data that’s coming in? If it’s new data coming in maybe setup a webhook instead on lessen the time intervals between the scenario runs.

6

Thanks @samliew those are good links - but I think they both assume being able to retrieve the entirety of the API call first - and that it is subsequent actions that fail due to the amount of bundles in the API call.
Actually what is happening is that the first action (the API call) is failing due to the file size. I am trying to retrieve the entire database to then do some filtering on before sending to Google Sheets as New Rows.

So my thoughts were to do an initial API call that just retrieves “Total Record Count”, then do subsequent API calls divided into “Total Record Count”/4. Or something like that. I actually have that working now. but it does seem cumbersome.

7

You still need a Repeater module.

Screenshot_2024-08-29_080831

Hope this helps! Let me know if there are any further questions or issues.

— @samliew

P.S.: Did you know, the concepts of about 70% of questions asked on this forum are already covered in the Make Academy. Investing some effort into it will save you lots of time and frustration using Make later!

8

Thanks @LinkYourTech I will give that a look.