Does the API support a filter that allows you to send data past a certain date? You can use the date Mark functions to do some addday() math and schedule it daily. It would help if you sent some details for the api call and some snapshots.
Sometimes the API call has a filter you can use on the call to return just the data you want. But if not yes you can use the route filter to process only orders within a certain time frame. Place the filter on the route after the api call.
@Elie start your scenario with a “search records” module of the data store app. Look for the latest record retrieved and make your GET call using that timestamp. Then you update your datastore with the latest record retrieved in your GET call. (you will have to create a datastore of course)
I managed to create my scenario but used 700+ operations in 1 go
HTTP get > Parse Jason > Filtter to 15 mins time stamp… the time stamp had to filter 700… it all worked and was brillaint… but this cant be viable because after 2 hours i would run out? I wasnt sure where to go from here
I wasnt should if I should I create the data store and how to go about this from there?
You could try to do it the other way around like @loic.wiseflow suggests. By using a datastore you can save the latest timestamp and orderID which you ran before. So the steps in your scenario would be:
Data Storage: search records. (find the latest record which holds the timestamp of last data you had before in a scenario run. This should contain atleast the timestamp, and you can also add the OrderID to also filter on this (to make sure it stops at that order))
HTTP: Get request (make the GET request on that API you are calling, and use the timestamp of the data storage to filter down the data. Then in a filter after this module you can also add the OrderID to filter to make sure this doesn’t go through.)
JSON: Parse JSON OR Aggregate JSON (I don’t know when you want to send the data to, but if the module supports an array of items you can also use the JSON aggregation. This only uses 1 operation and aggregates all the items into an array)
Whatever module to send it to
Hope this helps you develop further. If you are still stuck or have questions, please also share some screenshots since this provides a lot of information.