The scenario doesn’t run Immediately – instead, it runs once a minute because it uses an external API that has rate limitations.
But, the scenario does process 500 items from the Webhook queue at a time.
For each cycle, it processes the work and attempts to make an external API call using the HTTP module.
Most of the time, it works correctly. But occasionally, the upstream API call using the HTTP module will yield an error. In that scenario, I want to ROLLBACK just that one cycle and retry again. However, the current ROLLBACK logic will cause this job an any of the other earlier 499 jobs to also get reprocessed again (which is an absolute waste of time).
In this scenario, if the earlier 499 cycles are fine but the 500th cycle triggers a ROLLBACK, is it possible for the ROLLBACK to only apply to the last 500th cycle – rather than causing all earlier 499 cycles to get reprocessed again?
Yes, I know I can make the scenario only process one job (instead of 500) at a time, but that makes processing much slower because the fastest I can have a scenario run is once per minute – using the Immediate option isn’t possible due to other external constraints.
Are you using break module for this? I think that should solve the issue unless you want to do sequential processing of the webhooks that you are getting.
I tried using Break before, but the problem is that adding Break logic adds a minimum 1 minute delay to sequential processing. It’s actually faster to just push the failed job back onto the back of the Webhook queue.
So basically, where ever you see the error handling in Blue, I originally used ROLLBACK modules in those locations. However, when the webhook would process a batch of 250 items in the queue and let’s say it encounters an error in item #249 – then ALL previously successfully processed jobs would be “rolled back” and reprocessed again – in addition to the 249th job that failed – which is highly inefficient.
Instead, whenever I encounter an error in batch mode, I just shove the job back to the back of the queue and complete the cycle, which is much faster.
But, this feels like a hack. Ideally, the ROLLBACK module should be smart enough to offer 2 options:
ROLLBACK CYCLE ONLY (only the immediate job gets retried)
ROLLBACK ENTIRE EXECUTION (ALL of the jobs in the current execution get retried)
Can you show a screenshot of an actual run? To see how the bundles are processed and outputted? And maybe a history screenshot where you actually had the rollback issue?
screenshot of your make scenario for me, along with the relevant module configurations and 
share the images here?
