Read multiple lines of a CSV file and parse each line individually into columns

[utek.auto]

What are you trying to achieve?

My scenarios read a text file, normally it is a sing line of text in it with 7 values delimited by “|” character. so far I manage to extract each of the values with a parser module that passes it to an Iterator that separates each value and them the tools module allocate each to a variable. But sometime I have more than one line in the file and when I run the scenarios I get the only the first line values.
How can I get the parser to read each line separately and passes to iterator all lines as separated values to be processed?
Thank you.

Screenshots: scenario setup, module configuration, errors

[samliew]

Could you provide the contents of the file so we can use it as an example in a test scenario?

samliew – request private consultation

Join the Make Fans Discord server to chat with other makers!

[utek.auto]

SPF00292_31052023143349-DEMO.TXT (204 Bytes)

[samliew]

Welcome to the Make community!

Yes, that is possible. You’ll need a minimum of one module:

Output

Give it a go and let us know if you have any issues!

Module Export

You can copy and paste this module export into your scenario. This will paste the modules shown in my screenshots above.

1. Copy the JSON code below by clicking the copy button when you mouseover the top-right of the code block
   

2. Enter your scenario editor. Press ESC to close any dialogs. Press CTRLV (paste keyboard shortcut for Windows) to paste directly in the canvas.

3. Click on each imported module and save it for validation. You may be prompted to remap some variables and connections.

Click to Expand Module Export Code

JSON - Copy and Paste this directly in the scenario editor

{
    "subflows": [
        {
            "flow": [
                {
                    "id": 61,
                    "module": "util:ComposeTransformer",
                    "version": 1,
                    "parameters": {},
                    "mapper": {
                        "value": "00292|120230005|NETC1.2023.005-1|2987871|15520|20230720142551|PAID\n00292|120240406|NETC1.2023.406-1|2982234|39500|20240502142551|PAID\n00292|120230106|NETC1.2023.106-1|2980010|20520|20230612142551|PAID"
                    },
                    "metadata": {
                        "designer": {
                            "x": 3,
                            "y": -2225,
                            "name": "Text"
                        },
                        "restore": {},
                        "expect": [
                            {
                                "name": "value",
                                "type": "text",
                                "label": "Text"
                            }
                        ]
                    }
                },
                {
                    "id": 62,
                    "module": "csv:ParseCSV",
                    "version": 1,
                    "parameters": {
                        "colCount": null,
                        "csvContainsHeaders": false,
                        "delimiterType": "other",
                        "relax": false,
                        "delimiter": "|"
                    },
                    "mapper": {
                        "csv": "{{61.value}}"
                    },
                    "metadata": {
                        "designer": {
                            "x": 247,
                            "y": -2221,
                            "messages": [
                                {
                                    "category": "last",
                                    "severity": "warning",
                                    "message": "A transformer should not be the last module in the route."
                                }
                            ]
                        },
                        "restore": {
                            "parameters": {
                                "delimiterType": {
                                    "label": "Other"
                                }
                            }
                        },
                        "parameters": [
                            {
                                "name": "colCount",
                                "type": "number",
                                "label": "Number of columns"
                            },
                            {
                                "name": "csvContainsHeaders",
                                "type": "boolean",
                                "label": "CSV contains headers",
                                "required": true
                            },
                            {
                                "name": "delimiterType",
                                "type": "select",
                                "label": "Delimiter",
                                "required": true,
                                "validate": {
                                    "enum": [
                                        ",",
                                        "\t",
                                        "other"
                                    ]
                                }
                            },
                            {
                                "name": "relax",
                                "type": "boolean",
                                "label": "Preserve quotes inside unquoted field",
                                "required": true
                            },
                            {
                                "name": "delimiter",
                                "type": "text",
                                "label": "Delimiter character",
                                "validate": {
                                    "max": 1,
                                    "min": 1
                                },
                                "required": true
                            }
                        ],
                        "expect": [
                            {
                                "name": "csv",
                                "type": "text",
                                "label": "CSV",
                                "required": true
                            }
                        ]
                    }
                }
            ]
        }
    ],
    "metadata": {
        "version": 1
    }
}

samliew – request private consultation

Join the Make Fans Discord server to chat with other makers!

[samliew]

Alternative solution

You’ll need a minimum of two modules:

1. Iterate rows (split by newline)

2. Split cells by pipe character

Output

Give it a go and let us know if you have any issues!

Module Export

You can copy and paste this module export into your scenario. This will paste the modules shown in my screenshots above.

1. Copy the JSON code below by clicking the copy button when you mouseover the top-right of the code block
   

2. Enter your scenario editor. Press ESC to close any dialogs. Press CTRLV (paste keyboard shortcut for Windows) to paste directly in the canvas.

3. Click on each imported module and save it for validation. You may be prompted to remap some variables and connections.

Click to Expand Module Export Code

JSON - Copy and Paste this directly in the scenario editor

{
    "subflows": [
        {
            "flow": [
                {
                    "id": 57,
                    "module": "util:ComposeTransformer",
                    "version": 1,
                    "parameters": {},
                    "mapper": {
                        "value": "00292|120230005|NETC1.2023.005-1|2987871|15520|20230720142551|PAID\n00292|120240406|NETC1.2023.406-1|2982234|39500|20240502142551|PAID\n00292|120230106|NETC1.2023.106-1|2980010|20520|20230612142551|PAID"
                    },
                    "metadata": {
                        "designer": {
                            "x": 2,
                            "y": -2024,
                            "name": "Text"
                        },
                        "restore": {},
                        "expect": [
                            {
                                "name": "value",
                                "type": "text",
                                "label": "Text"
                            }
                        ]
                    }
                },
                {
                    "id": 58,
                    "module": "builtin:BasicFeeder",
                    "version": 1,
                    "parameters": {},
                    "mapper": {
                        "array": "{{split(57.value; newline)}}"
                    },
                    "metadata": {
                        "designer": {
                            "x": 245,
                            "y": -2025,
                            "name": "Iterate Lines"
                        },
                        "restore": {
                            "expect": {
                                "array": {
                                    "mode": "edit"
                                }
                            }
                        },
                        "expect": [
                            {
                                "name": "array",
                                "type": "array",
                                "label": "Array",
                                "mode": "edit",
                                "spec": []
                            }
                        ]
                    }
                },
                {
                    "id": 59,
                    "module": "util:SetVariable2",
                    "version": 1,
                    "parameters": {},
                    "mapper": {
                        "name": "values",
                        "scope": "roundtrip",
                        "value": "{{split(58.value; \"|\")}}"
                    },
                    "metadata": {
                        "designer": {
                            "x": 489,
                            "y": -2025,
                            "name": "Split pipe delimiter"
                        },
                        "restore": {
                            "expect": {
                                "scope": {
                                    "label": "One cycle"
                                }
                            }
                        },
                        "expect": [
                            {
                                "name": "name",
                                "type": "text",
                                "label": "Variable name",
                                "required": true
                            },
                            {
                                "name": "scope",
                                "type": "select",
                                "label": "Variable lifetime",
                                "required": true,
                                "validate": {
                                    "enum": [
                                        "roundtrip",
                                        "execution"
                                    ]
                                }
                            },
                            {
                                "name": "value",
                                "type": "any",
                                "label": "Variable value"
                            }
                        ],
                        "interface": [
                            {
                                "name": "values",
                                "label": "values",
                                "type": "any"
                            }
                        ]
                    }
                }
            ]
        }
    ],
    "metadata": {
        "version": 1
    }
}

samliew – request private consultation

Join the Make Fans Discord server to chat with other makers!