I am currently sending single requests to a webhook, which subsequently processes each request with an HTTP module, then this result is posted to Airtable.
I can get this to work for each individually, but the Make processing is getting expensive and I get throttled by Airtable when I have big spikes of requests. So I would like to batch up the result of the HTTP module into groups of 10 and then submit 10 at a time in the Airtable Batch Edit module. Ocassionally I might shoot through a number of requests not divisible by 10, so I also need the remainder to be pushed.
I consider myself an advanced (not expert) Make user (since 2013) and I’ve tried all kind of modules and configurations to make this work…repeaters simply repeat the same request; aggregators don’t seem to aggregate regardless of how I configure.
I can not submit a batch webhook and I prefer not to use a data store. I suspect there is an advanced operator or way to transform the data in a module that would make this work, but that is a level above my expertise. I would love any tips or help.
Hello @Keith_Pieper and welcome to the official Make Community!
So… I’m only going to say what you already know, but now that I’ve commented I’ll get to see what others have to say too.
You’re posting to a webhook, so your only option is to hold that data somewhere, followed by counting number of pending items, then process them when there are 10 or more. I don’t think there’s any way around it AND save ops.
Is there any chance you can more your existing scenario more efficient? Being a user as long as you have, I doubt it, but I’ve seen some ridiculously efficient, but extremely difficult to read, scenarios. Just a thought!
Welcome to the Make community!
I think you could just schedule the webhook instead of letting it run immediately.
That way requests will get stored in the webhook queue, and then you can process them in batches of up to your set amount, 10 (or less).
For further assistance, please provide the following:
1. Screenshots of module fields and filters, and any error messages
Please share screenshots of relevant error messages, module fields, and filters in question? It would really help other community members to see what you’re looking at.
You can upload images here using the Upload icon in the text editor:
2. Scenario blueprint
Please export the scenario blueprint file to allow others to view the mapped variables in the module fields. At the bottom of the scenario editor, you can click on the three dots to find the Export Blueprint menu item.
Hope this helps! Let me know if there are any further questions or issues.
— @samliew
P.S.: Did you know, the concepts of about 70% of questions asked on this forum are already covered in the Make Academy. Investing some effort into it will save you lots of time and frustration using Make later!
But then how do you batch them in batches of 10 to take advantage of Airtable batch update? Won’t the scenario run once for each queued webhook?
Correct me if I’m wrong, but Webhook triggers can also be scheduled if you want to queue incoming requests and process them together in the same batch (scenario execution).
This is covered in the Help Center https://www.make.com/en/help/tools/webhooks#processing-scheduled-webhooks and possibly the Make Academy too.
I’ve also mentioned it here, Troubleshooting Single-Cycle Issue in Telegram Bot Scenario - #2 by samliew
Hope this helps! Let me know if there are any further questions or issues.
— @samliew
P.S.: Did you know, the concepts of about 70% of questions asked on this forum are already covered in the Make Academy. Investing some effort into it will save you lots of time and frustration using Make later!
Nice never thought to try that!
Example
10 items in the queue
“Run Once”
Three output bundles when run
(based on 3 “Maximum number of results”)
You can aggregate this of course.
Queue down to 7
Hope this helps! Let me know if there are any further questions or issues.
— @samliew
P.S.: Did you know, the concepts of about 70% of questions asked on this forum are already covered in the Make Academy. Investing some effort into it will save you lots of time and frustration using Make later!
2 Likes
So I gave this some thought and experimentation. The challenge with most solutions is they end up needing to trigger on a schedule. Make only allows 1 minute scheduling which is too infrequent…immediate is the only feasible option.
What I have built is an Airtable that can be triggered by a view change to an Airtable module OR a script that pushes to a webhook. I have single record requests going out, but I also created a script that batches out 10 records into Make.
I have tested and created several options and this is the closest I can get…the challenge is while I pipe 10 records into the flow and the batch upload to Airtable is 1 operation with 10 records, the entire scenario runs 10 times (one per record) - the other 9 end up being duplicate runs. How can I get around this?
My current best
blueprint (5).json (138.7 KB)
Then you’ll have to make a feature request, as what I’ve shared is the only workaround.
You can submit this suggestion to the Idea exchange, under Platform ideas and improvements
Don’t forget to search for it first, just in case someone already suggested it, so that you don’t end up creating a duplicate.
Hope this helps! Let me know if there are any further questions or issues.
— @samliew
I made this work, though I needed to rebuild from scratch because Webhooks don’t support batch functionality like I assumed, whereas Airtable Search triggers do! I had all of my current scenarios built to trigger from webhooks to have a more module application that allowed me to build subprocess scenarios that could be called from numerous other scenarios, but it looks like Webhooks just doesn’t work like I assumed to process batch requests. https://www.make.com/en/platform-ideas/p/webhook-batch-support
2 Likes
