Using Data Store as a Data Buffer to Prevent Lost Webhook Data

Showcase
, ,
1

Working with webhook data…well it can be great because you get instant results.

It can also be bad, because if you don’t process that data when it’s sent by the webhook, it could be lost F O R E V E R !

Well, not forever, but it does require the use of Postman to retry / test with data that wasn’t fully processed.

That’s why I’ve made it one of my “Best Practices” to always use Data Store as a Webhook Data Buffer.

What is this gypsy magic you ask? Keep reading…

Below is a scenario where I watch QuickBooks Online for new events. When data is received from QBO, data is stored in the Data Store webhook buffer for future retrieval.

image
image804×812 57.6 KB

As you can see below, I’m saving the QBO Object ID, Type and Operation Type for the QBO asset. I’m also saving the QBO last modified time, current time and the execution log URL for this buffer record.

Note, I have “Overwrite Existing Record” set to Yes, because I don’t want to create any duplicate buffer records. This keeps everything unique.

image
image638×904 61.8 KB

Once the buffer record is saved, I use multiple HTTP modules to trigger the next scenario.

image
image804×812 60.4 KB

Next, you will have a companion scenario that will 1) start with a webhook, 2) search the data store buffer, then complete the steps that must be automated.

NOTE: Set your Data Store Search module to retrieve more than one (1) record if you want it to processed webhook data that has not been processed by your scenario. This harnesses the power of the webhook buffer. If a past webhook data package wasn’t processed, it can be attempted again when another bundles is sent through your scenarios!

image
image793×343 39 KB

Next, and this is important, add a Delete Data Store module at a place in your scenario where you no longer need your webhook buffer data. This ensures your data will not be processed a second time.

image
image1074×601 81.7 KB

Here’s a summary:

  • Scenario A has a webhook module as the trigger (this webhook contains API service data payload), and an action that saves the webhook data into a DataStore record. Also present is an HTTP module that fires the Scenario B webhook.

  • Scenario B starts with a webhook, searches for the saved Data Store Buffer records, then performs the necessary scenario actions, then ends by deleting the Data Store Buffer Record so it isn’t processed again.

If you get stuck, reach out and myself or a member of my team can help!

6 Likes
2

Whoa @andyoneil, how incredibly helpful! Thanks so much for sharing this nifty best practice of yours with the community, much appreciated :pray:

3

Clever! I assume you use the same store for all your scenarios since they’re keyed by the execution ID?

1 Like
4

Nice idea. Have you considered using Hookdeck? They will forward any webhooks from your source apps to Make, and retain a copy so they can easily be retried later if needed. No lost data!

5

We had a good TechTalk on Hookdeck, but Andy’s approach keeps it in make and I think it would be more flexible. As long as data store can grow as large as necessary.

1 Like
6

I don’t key them with the execution ID, I key them with the value that I’ll need in the second scenario.

7

Hey @PaulM, I wasn’t aware of Hookdeck, sounds like a powerful utility. I do like to keep as much as possible in Make.

8

@andyoneil, I get that. And I do most things in Make too, its been a terrific tool. I like Hookdeck because I can repeat a webhook (either for testing or because it failed and needs to run again to resolve the issue) and I don’t need anything different in Make. Just resent the webhook and it runs exactly as it would have earlier. Plus, I’ve got a few dozen webhooks now, integrating our various cloud platforms, so Hookdeck gives me a nice log of what’s happening and when - it’s a great dashboard of sorts for activity.

9

You can do that in make. Just make a scenario that reads the right key in the datastore and send it to http with the webhook as the url and the payload. There’s nothing really that special about hookdeck.

One thing it does do is introduce potentially more cost and another point of failure but that could be useful if make webhooks are down.

10

Great discussion. I strongly believe backing up webhook is necessary when the process is crytical.
With this in mind we released two modules on makemarket.

  • Webhook + Json returns both parsed and raw json so you can use it directly but also create a backup easy

  • Webhook - Hookdeck automatically creates a webhook and connects it to hookdeck. Also returns both parsed and raw json.

Feedback is highly appreciated.
Get an account here

image
image949×439 60.6 KB

1 Like