Check for non-existing data and adding to SmartSuite without using up all my credits?

[APeers]

I’ve been struggling with how to make my scenario work without using a ton of credits. Here is the basic workflow…

1. HTTP API call module which returns a list in an array
2. Iterate module
3. Add records to SmartSuite, only if they dont already exist

I’ve tried the SmartSuite Upsert module but that checks every single record in the table, same with the Search module.

I’ve also tried using the Data Store module but that also runs through ever row after the iterator, which uses a credit per data row.

I’m sure there’s a way to do this with an array but I’m just trying to figure out how to do that. If anyone has suggestions on how I could this while only using 1 credit per module (or close to) that would be great!

[damato]

Hi @apeers

The usual way to do this kind of stuff is with Bulk inserts. You iterate over the results and then aggregate them, setting as the target structure for the aggregator the module that is going to receive the array to insert in bulk.

This is the basic framework to avoid inserting item by item in any scenario that has a module that supports bulk inserts.

I’m not familiar with the Smartsuite app, but I see it does have a Bulk Create Records. The only problem is your requirement to only insert a record if it does not exist. You need to test it to see if it fits your use case.

@damato

[samliew]

Welcome to the Make community!

What damato said. Use an array aggregator with a bulk module.

Combining Bundles Using Aggregators

Every result (item/record) from trigger/iterator/list/search/match modules will output a bundle. This can result in multiple bundles, which then trigger multiple operations in future modules (one operation per bundle). To “combine” multiple bundles into a single variable, you’ll need to use an aggregator of some sort.

Aggregators are modules that accumulate multiple bundles into one single bundle. An example of a commonly-used aggregator module is the Array aggregator module. The next popular aggregator is the Text Aggregator which is very flexible and can apply to many use-cases like building of JSON, CSV, HTML.

You can find out more about the other types of aggregator modules here:

• Array aggregator
• Text aggregator
• Numeric aggregator
• Table aggregator
• JSON aggregator
• CSV aggregator
• Archive (ZIP) aggregator
• other app-specific aggregators

Question: Which is the best aggregator do you think you’ll need for your use-case?

Mapping a Specific Structure Into a Complex Field

If you have an array of collections, in programming terms, this is called an array of objects, or an array with non-primitive data types (“complex”).

The Array Aggregator module is very powerful because it allows you to build a new complex array of collections that matches a later module’s field to map multiple items (collections) to it. Such fields initially would allow you to manually add items, but you can toggle the “Map” switch to the “on” state and map a whole array into a single field.

This is done by selecting the “Target structure type” in an Array Aggregator module.

As you can see from the above example, the “Map” toggle on complex fields are used when you have an array variable (like from an array aggregator). Other combinations of modules may also allow you to generate an array that matches a future field’s array structure, like “Aggregate to JSON + Parse JSON”, or “Create JSON + Parse JSON”, but this is an advanced topic.

Question: Are you mapping your array into a field that accepts more than one item/collection?

Example

Here is an example of how your scenario could look like:

This is just an example. Your final solution may or may not look like this depending on your requirements and actual data.

For more information, see “Mapping with arrays” in the Help Centre. You should also do the Make Academy, which also covers the use of Iterators & Aggregators.

Hope this helps! Let me know if there are any further questions or issues.

— @samliew
_{P.S.: investing some effort into the tutorials in the Make Academy will save you lots of time and frustration using Make!}

[APeers]

Would using the Smartsuite Find Records module work for my purposes? The Comparison Value field does seem to accept arrays, but when I add the array from the Array aggregator, it doesnt seem to pull anything meaningful - meaning the conditional filter is not working.

My thought process here is that I could search the Part No. filed in Smartsuite and compare it to the “number” field in the Array. It doesnt error out, but it’s pulling data that already exists, not data that doesnt exist.

Maybe, I’m not grasping these concepts, but I’m racking my brain here trying to figure this out.