Converting object keys to an JSON array

[cawsanders]

Goal with this scenario.

• I’m working on getting a series of keys on a JSON webhook input into an email template.
• As some of the keys may be empty, I can’t just pass them to the MailGun email template or it will complain the value is missing.
• I was able to get it working with a custom JSON structure for the mail variables that I parse and iterate on. I’m not using all the variables I need in this example.
  
  
  
  
• I’m then iterating on the array object that is created from the previously mapped JSON keys and aggregating it into a MailGun template variables destination type while filtering out any keys that may not have a value. Required since some source events do not contain that data, while others do.
  
  
  
  
• Mailgun is taking the output aggregate array and is happy.

Request

Is this the correct way to do this? Or is there a more efficient / optimized way to handle this that doesn’t require creating and re-parsing a JSON object into a different format than what the webhook recevies?

[Donald_Mitchell]

Hello @cawsanders and welcome!

With Mailgun, are you saying if a value is blank/“”/empty string, then it’s not allowed to be submitted? If so, you might see if it will allow NULL instead and the ifempty() function can help you.

{{ifempty(value; null)}}

This basically just says if value is empty, then use null instead. null is slightly different than empty string.

Are you able to post an example of the error Mailgun gives you when you supply a bad JSON and what that JSON looks like?

[cawsanders]

I’m using the make integration.

Thanks for the ifempty suggestion, I did try that prior to posting with the same result.

I was manually mapping the keys and values to the template variables.

If I supply the template key-value value with null, emptystring, or empty object it rejects the request saying that 1 or more values are missing (depending on how many template variables were empty).

If I use the method I posted here it works as the null-value keys are not sent, but feels clunky. I was wondering if there was a better way to extract the values from the source webhook bundle that is parsed to JSON and map them instead of what I did here.

[Terry_Hopper]

Hi @cawsanders

Sometimes there is no “Correct” or incorrect way. If it is working, then great!

However, if you’re trying to optimise to reduce operations, have you tried the toArray() function? This might save you mapping into your custom JSON object, and should give you an array that you can iterate and filter.
The article here may help: https://www.make.com/en/help/functions/array-functions

Without seeing the data coming into your webhook, I can’t be sure this would work for you. If you need more help, it would be great if you could also screenshot a bundle received by the webhook.

Good luck!

[cawsanders]

Thank you everyone taking the time to look at this. Not sure if I can do the toArray directly on the parsed webhook object.

For the scenario, we get it as a webhook (JSON string), then parse that to an object via Parse JSON module.

Webhook Module Bundle Input

This is what is present in the value field on the webhook, in expanded form

{
  "alarm_destinations": ["{GUID}"],
  "rule_strategy": "{STRING}",
  "packet_type": "{STRING}",
  "timestamp_received": "{UNIXMILLIS}",
  "rule_id": "{STRING}",
  "suppressed": "{STRING}",
  "sensor_uuid": "{GUID}",
  "x_att_tenantid": "{GUID}",
  "rule_dictionary": "{STRING}",
  "alarm_response_codes": [],
  "rule_attack_technique": "{STRING}",
  "timestamp_arrived": "{UNIXMILLIS}",
  "priority": "{STRING}",
  "alarm_source_asset_ids": ["{GUID}"],
  "packet_data": [],
  "destination_name": "{GUID}",
  "destination_asset_id": "{GUID}",
  "destination_canonical": "{GUID}",
  "alarm_events_count": 0,
  "x_att_tenant_subdomain": "{STRING}",
  "alarm_sensor_sources": ["{GUID}"],
  "alarm_destination_assset_ids": ["{GUID}"],
  "needs_enrichment": false,
  "priority_label": "{STRING}",
  "timestamp_occured": "{UNIXMILLIS}",
  "needs_internal_enrichment": false,
  "events": [{}],
  "rule_attack_tactic": ["{STRING}"],
  "rule_intent": "{STRING}",
  "source_canonical": "{GUID}",
  "highlight_fields": [
    "destination_username",
    "source_ntdomain",
    "destination_canonical",
    "source_canonical",
    "audit_reason",
    "rule_attack_id",
    "rule_attack_tactic",
    "rule_attack_technique"
  ],
  "event_type": "{STRING}",
  "alarm_sources": ["{GUID}"],
  "uuid": "{GUID}",
  "rule_attack_id": "{STRING}",
  "transient": false,
  "source_asset_id": "{GUID}",
  "source_name": "{GUID}",
  "audit_reason": "{STRING}}",
  "status": "{STRING}",
  "rule_method": "{STRING}",
  "destination_username": "{STRING}",
  "timestamp_to_storage": "{UNIXMILLIS}"
}

Parse JSON Output - initial

I don’t have a collection reference that is usable in a method like toArray, unless the Bundle root collection can be used to transform to an array of keys.

[Terry_Hopper]

Hi @cawsanders ,

I don’t have a collection reference that is usable in a method like toArray, unless the Bundle root collection can be used to transform to an array of keys.

I think you’ve hit on the solution!

1. In your “Parse Alarm JSON” module, change the JSON string field. Just paste this in:

{“data”: {{1.value}}}

This will create a “root” element called “data” that holds the collection, which you can use as a handle in the next steps.

2. In your iterator, refer to this handle and convert it to an array. Just paste this in:

{{toArray(3.data)}}

Let us know how it goes.

[cawsanders]

Looks promising, thank you! I’ll explore that direction.

[cawsanders]

I’ve run into a related issue.

The toArray() on the data element or the raw array object in that data element will now let me iterate over the keys and pick what I want, ending up with key, value output at the iterator. Also makes it easier to apply many of the array/collection functions. That was super helpful.

Have you run into this before? The keys and values are exactly what I want, but the key is not available for aggregation.

Iterator input.

Iterator output

Available variables after a test run to get data parsed.

Array aggregator / JSON aggregator, etc. The key field is missing.

If I use a variable / string, etc. module in between, I can see that it does exist.
It does correctly pull out the key in the string module as well, even though Make doesn’t show it as an available property. Key was “rule_attack_technique”

If I attempt to select the output of the string module, it gets unselected upon save. I’m confused, because this looks like it is intended to be able to get the key from the iterator via aggregation.

I did notice that if I use {{44.key}} as the group by clause. I get returned a collection close to the key, value format I want, but it would require one additional operation.

I also noticed that with the Aggregate to JSON module, I get a JSON string that is what I want but requires another parse operation.

I ideally want just the array output containing a series of collection objects with a Key/Name and Value, so I could merge other template variables who require different logic to create.

The Mailgun module template variables takes an Array of Collections with a Name and Value key.
In addition to what’s outlined below, I have other data I’d want to merge into that array of objects who require different functions applied to them.
I’m currently doing this with a bunch of variables, but I need to somehow apply the functions to the key and then merge with the output for the key, value iterator for emails or storing as JSON to re-use.

Kind-of fixed the issue, but posting so others can see the behavior I observed
Final array aggregator ended up showing the key field. Initially, this field was not visible no matter how many runs I went through, so I was having a hard time getting it to output in the desired format without an additional operation. The key field became available right as I was creating this post and gathering screenshots.

[cawsanders]

I was thinking that I could get the list of keys in addition to what I’m already selecting into the same iterator, then use a router to filter on any keys with special transformations required.

See image for layout.

Not sure if it would be better to have the gathering iterator include the keys requiring the transformations, then have a router between it and the aggregator, I don’t think it behaves that way.

If both of the arrays come out looking like this, I should be able to merge them via the merge() operator, correct?

[Terry_Hopper]

Hi @cawsanders ,

If you need to do transformations, my approach would definitely be to do them before you aggregate the array. You then won’t need to merge the arrays.

But you can’t put a router between an iterator and aggregator. If you only have a few of the values that will need transforming, you could add a Set Variable module in between your iterator and aggregator. Using switch() or if() formulas instead of a router, you can replace just certain key/value pairs while leaving the others intact.

If you have more complex transformations, I would explore the idea of creating an additional scenario to handle this and then send the key/value to the scenario webhook using an HTTP module in between the iterator and aggregator, and then use the webhook response as the value to aggregate.

Let us know how it goes!

[cawsanders]

Thank you for the advice.

I have played with chained scenarios, but the current concern is that I need a way to ensure that the chained webhook cannot be called outside of our make organization due to the data being transmitted.

This is the current child scenario:

I’d like to use the Run scenario action and then pull from adata store.
However, without a sleep module, the data store record does not exist yet.

This works.

Without the sleep fails, as the record does not exist yet.

In the event data takes longer to process, the sleep module may fail.

Goal would be to use the run scenario module, validate successful completion, and then check the result from a data store.

I’ve seen it mentioned that an array aggregator can be used to ensure successful bundle execution, but that didn’t seem to work How to Wait for an Operation to Complete Before Proceeding?.

I might be able to use the raw API to do this with the responsive flag, I think.

[Terry_Hopper]

Hi @cawsanders ,

I would still recommend you use an HTTP module to call a webhook in your second scenario. This will handle the waiting condition for you.

If security is a concern, you can add security to the second webhook, so that it only returns data if a unique key is passed in the webhook headers—a unique key that only your organisation knows. You can create this key as a global variable and pass it through the HTTP module from the calling function, and check it from a filter straight after the webhook in the secondary scenario… This will be so much more robust than guessing the time it will take for a scenario to finish. For example, what if the scenario stops with an error, you might not catch this at all?

You could even use a router in your second scenario to return a 401 “Unauthorized” status to any call that doesn’t include your unique key in the headers.

You can also add IP filtering, so that only calls from Make itself are allowed to connect to your webhook. The IP addresses are listed here.
https://www.make.com/en/help/connections/allowing-connections-to-and-from-make-ip-addresses

[Terry_Hopper]

@cawsanders

I might be able to use the raw API to do this with the responsive flag, I think.

…but yes, you can also use the Make API. You’ll need to create an API key, and connection to Make, but then you can use the “Make an API Call” module from the built-in Make app to run your scenario and wait for your response. Do beware, you will get a timeout error if the scenario takes longer than 40 seconds to respond.

[cawsanders]

So far that appears to work great.