Hello,
I’ve developed a TikTok scraper to analyze the performance of TikTok videos. This scenario involves scraping all the videos of a creator, along with information such as views.
The issue is that it incurs a high number of operations, which becomes costly over time, especially when repeatedly searching through a database like Airtable to check if a record already exists. For example:
Is there a way to optimize this scenario to reduce the number of operations and associated costs?
Welcome to the Make community!
I’m not sure how you get 293, unless it’s ALL the 293 videos of a single account.
You can use a dedicated scraper that returns you ALL the videos details in a single HTTP request.
Try looking for one on RapidAPI.
Yes, I am using RapidApi.
Here is the Airtable search:
It always searches if the record is already in the database so it’s not duplicated.
It’s not duplicated but like you said,
so perhaps you might want to aggregate the IDs into a single array and see if you can perform the 293 searches in a single operation, in Airtable.
If that’s not possible, you might want to find and switch to some other service that you can search an array of values in a single range.
