So, basically what my scenario is doing is accessing 3 different large CSVs that contain hundreds of thousands of lines each. The scenario is triggered by a webhook in DocuWare. When an invoice is stored in DocuWare and hits a specific point in the AP process, the scenario is triggered to access these 3 CSVs one by one and grab specific PO information from the CSVs that are related to the invoice being processed. The goal is to use Make to index PO data from these 3 CSVs into a table on the invoice in DocuWare. It works great with dummy CSVs that only have like 100 lines.
But, when I try the scenario with the real CSVs, they are so big that the scenario basically get’s hung up trying to access them and then freezes. My question is, is this the most efficient way to do this? Is there another way? I feel like the core of the scenario, the indexing into DocuWare is fine. But, is there a better way to access these CSVs thatn the FTP module? For the record, with a DocuWare cloud system, you get an FTP site to store information and so we store the CSVs that need to be accessed there. Make then wil use the FTP - Get a file module to access the CSVs, and then search the CSVs for a matching invoice and store number.
I know that it’s getting hung up because the scenario is basically going through a huge CSV file every time it runs. So, is there a more efficient way to do this that anybody can think of? Any help or suggestions would be much appreciated. And I can offer more information as well if I need to.
I’ve included some screen shots below and also 1 line examples of the 3 CSVs the scenario accesses.
