FTP Module to Access Large CSVs - Issues

So, basically what my scenario is doing is accessing 3 different large CSVs that contain hundreds of thousands of lines each. The scenario is triggered by a webhook in DocuWare. When an invoice is stored in DocuWare and hits a specific point in the AP process, the scenario is triggered to access these 3 CSVs one by one and grab specific PO information from the CSVs that are related to the invoice being processed. The goal is to use Make to index PO data from these 3 CSVs into a table on the invoice in DocuWare. It works great with dummy CSVs that only have like 100 lines.

But, when I try the scenario with the real CSVs, they are so big that the scenario basically get’s hung up trying to access them and then freezes. My question is, is this the most efficient way to do this? Is there another way? I feel like the core of the scenario, the indexing into DocuWare is fine. But, is there a better way to access these CSVs thatn the FTP module? For the record, with a DocuWare cloud system, you get an FTP site to store information and so we store the CSVs that need to be accessed there. Make then wil use the FTP - Get a file module to access the CSVs, and then search the CSVs for a matching invoice and store number.

I know that it’s getting hung up because the scenario is basically going through a huge CSV file every time it runs. So, is there a more efficient way to do this that anybody can think of? Any help or suggestions would be much appreciated. And I can offer more information as well if I need to.

I’ve included some screen shots below and also 1 line examples of the 3 CSVs the scenario accesses.




FTP Module

Hi! Welcome to the Make community!

I know by experience that the Parse CSV module can be blocked when there are too many lines to process. It’s not the FTP that causes troubles I believe…

I’m not sure it can work properly, but you could give a try with Regex to pick the line that corresponds to the relevant data (Vendor Number or Inv Number, …), and then it will be easy to parse that line using Split, map, get, functions.

Can you try and use Text Parser / Match Pattern to extract the relevant line?
Like this you don’t need to parse the full CSV.

If you need help for the regex, let me know

Benjamin

2 Likes

Thanks! I will definitely try this. I will give it a go first, but I may need more guidance. Thanks again!

2 Likes