I’m currently stuck with a scenario that is supposed to fetch data of an API.
When I make the API call, I’ll get the first 50 entries of the desired data. To get the other 50 entries, I will have to repeat this call (e.g https://API-Endpoint/?page=1, https://API-Endpoint/?page=2, etc.).
In my scenario I added an Router module to check, whether I get an empty response or not.
If not, an Iterator module is supposed to write the data in a CSV-File. If my response is empty, I end this scenario.
Unfortunately, I don’t get it to work.
I set the repeater to 5 repeats. In the following screenshot you can see, that the Iterator is called 3 times but never proceeds to writing the CSV-File. As soon as I get an empty response (fourth repeat), the flow stops working, which is logical, because of the “Commit” Module.
My goal is, that the scenario writes the CSV-File and do the upload. After that, the scenario can stop.
Something wrong with the mappings in iterator that its not generating output to process further. lease share the screenshot of mappings in iterator module.
Regards, Msquare Automation - Gold Partner of Make
Hello @david_schkalej,
Typically, the response from the API should give you enough info to figure out exactly how many times you need to repeat the call to grab multiple pages, is this not the case for you?
By any chance are you able to share the API endpoint or the output from the first page?
Hi @Donald_Mitchell, thanks again for your response. Unfortunately, I don’t get the exact information about how many repeats that are necessary.
I receive the key “next”. If this key is not empty, I know that there is another page. If it is empty, I know that I’m on the last page.
In the following screenshot you can see the endpoint (it’s not an open API) as well as the output from the first page.
Thanks for those details.
Just curious, what happens when you request a page like 100 that you know will not exist?
I’m wondering if that will return a value for previous, maybe the the actual last page?
Just thinking a little outside the box here!
Also, I had a question about your Parse JSON #181 module. Are you parsing the return from HTTP? Usually HTTP can parse without requiring a separate Parse module, but there are some circumstances it might be required to separate the two functions.
Thanks for your answer. I really appreciate your help on this!
To identify what the last page is, there is no other way than checking if key “next” is not empty.
If I’m on page 5 (doesn’t exist), I will get the following output:
As you can see, next is empty. Unfortunately, previous is not empty. The previous value (page=4) also doesn’t exist.
But you can also see that the array “list” is also empty, because it’s an empty page.
I configured the JSON module, because I had some issues when parsing the response with the http module, but it’s probably not necessary. It was just easier when I configured the scenario.
Repeater, starting with 1 and repeat a reasonable number of times →
HTTP Make a Request (Page = Repeater #, Parse JSON On) →
Router →
Route 1 → If successfully parsed (List Array exists) → Tools Get Variable (Get Final Array) → Set Variable (Final Array merged with newly-received array)
Route 2 → If unsuccessful (List Array empty) → Tools Get Variable (Get Final Array) → Proceed with remaining steps of the scenario
The first time you “Get Final Array” the variable will be empty and return nothing.
Merge Result to Final Array will use merge() function to combine nothing with the List array from the HTTP module.
Next time route 1 runs, it will get the Final Array variable, merge it with the List Array from HTTP then overwrite the Final Array variable with the new combined array.
At the end of your scenario (at the end of route 2), you need to end the scenario before it continues repeating and taking route 2 again.
You saw this in the original post your referenced where they mentioned an Error Creator HTTP Module followed by a Commit module on Error Handler Route.
