Using Data Store as a Data Buffer to Prevent Lost Webhook Data

[andyoneil]

Working with webhook data…well it can be great because you get instant results.

It can also be bad, because if you don’t process that data when it’s sent by the webhook, it could be lost F O R E V E R !

Well, not forever, but it does require the use of Postman to retry / test with data that wasn’t fully processed.

That’s why I’ve made it one of my “Best Practices” to always use Data Store as a Webhook Data Buffer.

What is this gypsy magic you ask? Keep reading…

Below is a scenario where I watch QuickBooks Online for new events. When data is received from QBO, data is stored in the Data Store webhook buffer for future retrieval.

As you can see below, I’m saving the QBO Object ID, Type and Operation Type for the QBO asset. I’m also saving the QBO last modified time, current time and the execution log URL for this buffer record.

Note, I have “Overwrite Existing Record” set to Yes, because I don’t want to create any duplicate buffer records. This keeps everything unique.

Once the buffer record is saved, I use multiple HTTP modules to trigger the next scenario.

Next, you will have a companion scenario that will 1) start with a webhook, 2) search the data store buffer, then complete the steps that must be automated.

NOTE: Set your Data Store Search module to retrieve more than one (1) record if you want it to processed webhook data that has not been processed by your scenario. This harnesses the power of the webhook buffer. If a past webhook data package wasn’t processed, it can be attempted again when another bundles is sent through your scenarios!

Next, and this is important, add a Delete Data Store module at a place in your scenario where you no longer need your webhook buffer data. This ensures your data will not be processed a second time.

Here’s a summary:

• Scenario A has a webhook module as the trigger (this webhook contains API service data payload), and an action that saves the webhook data into a DataStore record. Also present is an HTTP module that fires the Scenario B webhook.

• Scenario B starts with a webhook, searches for the saved Data Store Buffer records, then performs the necessary scenario actions, then ends by deleting the Data Store Buffer Record so it isn’t processed again.

If you get stuck, reach out and myself or a member of my team can help!

[Michaela]

Whoa @andyoneil, how incredibly helpful! Thanks so much for sharing this nifty best practice of yours with the community, much appreciated

[alex.newpath]

Clever! I assume you use the same store for all your scenarios since they’re keyed by the execution ID?

[PaulM]

Nice idea. Have you considered using Hookdeck? They will forward any webhooks from your source apps to Make, and retain a copy so they can easily be retried later if needed. No lost data!

[alex.newpath]

We had a good TechTalk on Hookdeck, but Andy’s approach keeps it in make and I think it would be more flexible. As long as data store can grow as large as necessary.

[andyoneil]

I don’t key them with the execution ID, I key them with the value that I’ll need in the second scenario.

[andyoneil]

Hey @PaulM, I wasn’t aware of Hookdeck, sounds like a powerful utility. I do like to keep as much as possible in Make.

[PaulM]

@andyoneil, I get that. And I do most things in Make too, its been a terrific tool. I like Hookdeck because I can repeat a webhook (either for testing or because it failed and needs to run again to resolve the issue) and I don’t need anything different in Make. Just resent the webhook and it runs exactly as it would have earlier. Plus, I’ve got a few dozen webhooks now, integrating our various cloud platforms, so Hookdeck gives me a nice log of what’s happening and when - it’s a great dashboard of sorts for activity.

[alex.newpath]

You can do that in make. Just make a scenario that reads the right key in the datastore and send it to http with the webhook as the url and the payload. There’s nothing really that special about hookdeck.

One thing it does do is introduce potentially more cost and another point of failure but that could be useful if make webhooks are down.

[makeitfuture]

Great discussion. I strongly believe backing up webhook is necessary when the process is crytical.
With this in mind we released two modules on makemarket.

• Webhook + Json returns both parsed and raw json so you can use it directly but also create a backup easy

• Webhook - Hookdeck automatically creates a webhook and connects it to hookdeck. Also returns both parsed and raw json.

Feedback is highly appreciated.
Get an account here

[Robert_McKay]

Very cool, thanks @andyoneil. Actually thinking I could implement this to solve an issue in the CRM workflow you walked me through a few weeks ago on a consult call. The idea was to have scenarios watching each app where new customers potentially enter the business (schedulers, webforms, email subs, etc.), end each of those with an HTTP request that rationalizes the data from each app into an identical format, then receive that HTTP with a webhook-triggered companion scenario that checks all the other data sources and updates or creates records in each. Here’s a simplified sketch of the process with only two apps, A and B:

But as you can see, the issue is that if a customer signs up via app A, and wasn’t yet in app B, then the second scenario will create a record for them in B, which will trigger the scenario watching for new records in B, which will trigger the second scenario again, wasting operations on unnecessary updates of both records with no new data.

I can’t really think of a way to prevent new records triggering their respective watcher scenarios, but I was thinking I could store new records in a data store and then check that data store right after the webhook, stopping the scenario if it’s already in the data store.

In the above screenshot I have that happening at (2), but I realize I could actually do it in the watcher scenarios and prevent them even making the HTTP requests.

What do you think of this?

[andyoneil]

It sounds like a solid solution. Keep in mind you might get a buildup of unneeded Data Store records, sounds like you’ll need a way to handle that as well.

[Joseph_Accountant]

@andyoneil thank you for sharing, putting webhook payloads into a data store is a great idea and I will probably start implementing a similar solution in some of my scenarios.

I forget when the ‘Make’ modules became available/robust, but I think I would use the ‘Make’ module for ‘run a scenario’ now instead of the HTTP calls you had implemented in your original solution. So I would pass the data store ‘key’ to the ‘run a scenario’ module, and then the next scenario being run would have to be set to ‘on demand’ and would have to accept the scenario input for the data store ‘key.’

Can you think of any downside to this approach vs the HTTP + Webhook approach?

Anyways, thank you again for sharing, and I agree that this is a great use of the data store, and webhook data can be a pain to ‘repush’ for testing, so this is a great solution in many uses.

[andyoneil]

@Joseph_Accountant I think that sounds like a good solution. I haven’t used the “Run a Scenario” module yet, but I believe you’ll get the same result.