Help with Overcoming the 40-Minute Time Limit on Large PDF Processing Workflow

How To
1

Good Morning,

I’ve been exploring various community posts on the 40-45 minute time limit issue but haven’t found a clear solution that fits my workflow. I understand that the best way to prevent hitting this limit is to move the longest-running section outside the current scenario and call it as needed, but I’m unsure exactly where and how to implement this in my scenario.

Background:
I’m working with a very large PDF file from Google Drive, which I then process through PDF.co to separate it into different invoice groups. After that, I use PDF.co’s job check function to retrieve hundreds of individual files. My iterator breaks them out to set multiple variables to match the number of extracted files, after which I call PDF.co again to pull specific invoice data from each individual file. Finally, I attach files to emails, generate the subject and body from the data, etc.

The issue I’m encountering is the sheer volume of files I need to send through the PDF.co parser. Although PDF.co works quickly, the volume of data is causing me to exceed the 40-minute time limit.

What I’m Trying to Solve:
I want to figure out a way to break up this large volume of work so that I don’t exceed the time limit. Can anyone help me figure out how to efficiently break this process into smaller chunks or rework my scenario to avoid hitting the time limit?

Any advice on where I can split up the processing or optimize this workflow would be greatly appreciated!

Heres is the blue print
blueprint (5).json (77.2 KB)

2

Hey Wade,

just to clarify - you didn’t find a clear solution, because there isn’t one. Every solution to overcome the 40 min timeout is custom and highly dependent on the flow it self.

In your case I suggest one scenario that makes the initial call to PDF.co and stores all the results in a Make data store, then makes an HTTP call to start a second scenario. The second scenario grabs X amount of items from the data store and processes them, then checks if there are more items in the store to be processed → if there are, the scenario retriggers itself.

1 Like
3

Thanks for the suggestion! I didn’t think about breaking up the processing like that. It makes sense to store the individual invoices in Make’s data storage and then process them in smaller chunks, running for something like 30 minutes at a time.

I’ve used Make’s data storage before, so that part is clear. But I’m not sure how to limit the number of operations or the run time in Step 2. Here’s what I’m imagining:

  1. Run the process to break out all the files and create records in Make’s data storage.
  2. Activate the processing scenario, limiting it to 30 minutes of processing time. As each item is processed, I’d delete the corresponding entry from data storage since it’s no longer needed.
  3. Check for remaining entries in storage and continue the process until all items are handled.

My main questions are:

  • How would I call Step 2 after Step 1 is completed?
  • If I add a “Call Another Scenario” function, will starting the new scenario reset the time limit, or will it continue running under the same time restrictions?
  • How can I specifically limit the run time of Step 2 to 30 minutes? Do I need to set up a timer or control the flow in a different way?

Any advice or tips on how to set this up properly would be hugely helpful!

4

I don’t think you can limit it by time. Check how long processing one invoice takes and see how many you can do in 30 mins. Say if it takes 1 min per invoice then have the scenario only grab 30 invoices and then be done.

And yes, staring the scenario again will start a different timer.

5

Perfect, Thank you!!!