Selective Rollback Possible?

Hi There,

I’ve got a webhook-triggered scenario.

The scenario doesn’t run Immediately – instead, it runs once a minute because it uses an external API that has rate limitations.

But, the scenario does process 500 items from the Webhook queue at a time.

For each cycle, it processes the work and attempts to make an external API call using the HTTP module.

Most of the time, it works correctly. But occasionally, the upstream API call using the HTTP module will yield an error. In that scenario, I want to ROLLBACK just that one cycle and retry again. However, the current ROLLBACK logic will cause this job an any of the other earlier 499 jobs to also get reprocessed again (which is an absolute waste of time).

In this scenario, if the earlier 499 cycles are fine but the 500th cycle triggers a ROLLBACK, is it possible for the ROLLBACK to only apply to the last 500th cycle – rather than causing all earlier 499 cycles to get reprocessed again?

Yes, I know I can make the scenario only process one job (instead of 500) at a time, but that makes processing much slower because the fastest I can have a scenario run is once per minute – using the Immediate option isn’t possible due to other external constraints.

Alright, I think I figured out a workaround that doesn’t sacrifice throughput:

  • When processing a given cycle…
  • If the HTTP error throws an error…
  • Catch the error and then shove that one cycle’s job back onto the webhook’s queue…
  • And mark the exception as IGNORE.

This seems to be a decent workaround, but convoluted…

Hmm, that Sounds strange.

Are you using break module for this? I think that should solve the issue unless you want to do sequential processing of the webhooks that you are getting.

This sounds like the rollback is setup on the wrong module @dkindlund .

Can you take a camera-viewfinder-duotone screenshot of your make scenario for me, along with the relevant module configurations and share-all-duotoneshare the images here?

I tried using Break before, but the problem is that adding Break logic adds a minimum 1 minute delay to sequential processing. It’s actually faster to just push the failed job back onto the back of the Webhook queue.

So basically, where ever you see the error handling in Blue, I originally used ROLLBACK modules in those locations. However, when the webhook would process a batch of 250 items in the queue and let’s say it encounters an error in item #249 – then ALL previously successfully processed jobs would be “rolled back” and reprocessed again – in addition to the 249th job that failed – which is highly inefficient.

Instead, whenever I encounter an error in batch mode, I just shove the job back to the back of the queue and complete the cycle, which is much faster.

But, this feels like a hack. Ideally, the ROLLBACK module should be smart enough to offer 2 options:

  • ROLLBACK CYCLE ONLY (only the immediate job gets retried)
  • ROLLBACK ENTIRE EXECUTION (ALL of the jobs in the current execution get retried)

^ But this flexibility doesn’t currently exist.

Can you show a screenshot of an actual run? To see how the bundles are processed and outputted? And maybe a history screenshot where you actually had the rollback issue?

1 Like