Transform API Response from Collection of Collections to Array of Collections for Custom Module

Questions & Answers
1

Hi Makers,

I’m experimenting with creating a custom module (App) that consumes the Alpha Vantage API.

The API’s “Intraday” time series route returns its data points as structure of nested collections - here’s a link to their demo call. I’m not a fan of how they’re structuring the response for this route because

  1. Each data point is structured identically, so I think they should have been returned as an array of collections rather than a nested set of collections.
  2. They’ve assigned each sub-collection a key containing the date/time of the data point - I think it would have been better to return “dateTime”: “2022-07-14 18:45:00”.

I’d like to abstract away these idiosyncrasies in my module to make the data easier to consume in downstream modules. Ideally I’m trying to make the module return one bundle containing a collection for the metadata returned from the route and an array of collections containing the time series data points’ dateTime, open, high, low, close, and volume.

I tried working in the Interface tab of the module, and also played around with the “output” setting in Communication. I also looked into the IML function toArray but couldn’t figure out where I might inject this to turn the time series nested collection into an array of collections.

It would be great if you could point me in the right direction for how to solve this! :smiley:

Thank you!
John

2

Hi John, could you share an example output from the API? That might help us provide some direction quickly.

3

Thanks for the response Darin! Of course - attached is the output from the API call. It’s also available via this live demo call from the API: https://www.alphavantage.co/query?function=TIME_SERIES_INTRADAY&symbol=IBM&interval=5min&apikey=demo

response.json (22.3 KB)

As is, my custom module is producing output in this structure:

Bundle 1 (collection)
   Meta Data (collection)
   Time Series (collection)
      2022-07-14 18:45:00 (collection)
      2022-07-14 16:25:00 (collection)
      2022-07-14 16:15:00 (collection)
      ...

Because of the way the API was designed, each of the time series data points are sub-collections, with a key that is data. The API would have been a lot easier to consume had the time series been expressed as an array, but this experiment has given me a good chance to learn Make and exercise its flexibility :smiley:.

I understand I could let my module output as-is and leave it to the user to use iterators, repeaters, etc. to process the time series data downstream, but this would generate many operations so what I’m after is for my custom module to generate output in this structure:

Bundle 1 (collection)
   Meta Data (collection)
   Time Series (*ARRAY*)
      Intraday (collection)
      Intraday (collection)
      Intraday (collection)
      ...

Each Intraday would look like this for example:

Intraday (collection)
   Date and Time: 2022-07-14 18:45:00,   // note the key is transformed to data
   Open: 139.24,
   High: 139.25,
   Low: 139.24,
   Close: 139.25,
   Volume: 259

I feel like the answer (if this is possible) is somewhere in Interface and/or Communication > Output, maybe including use of toArray, but I can’t seem to hit on it.

Thank you for any advice or direction!
John

4

Hi John,

Have you been able to find a solution to this?

Cheers :slight_smile:

5

Hi Beaver,

Unfortunately no, but I’ll admit I’ve put this aside for the last little bit. I imagine this is somewhat an edge case so unless there’s an out-of-the-box way to do it I would likely solve it by standing up a converter API endpoint somewhere (maybe using node.js, Python, or a cloud tool) that consumes the AlphaAdvantage API and converts the data to a more usable response. Depends on whether the cost of running the converter API is worth the savings in operations on the Make side.

I still feel like there’s a built-in way to do this, but haven’t had the time to get back into it.

Regards,
John

1 Like
6

Hi @John_Rager,

I don’t know how to do this with a custom app but it seems to work with toArray() in Make (so it should be possible with a custom app):

image
image982×646 48.2 KB

image
image931×973 60.8 KB

2 Likes
7

Thank you for this lead @loic.wiseflow! - looks promising. I’ll see if I can work toArray into the custom app somehow.

8

I don’t know the specifications of your custom app but you can use IML functions inside the code! :slight_smile:

2 Likes
9

Hi @loic.wiseflow. That was the answer! I had to do a bit of extra funky stuff to get around the fact that AlphaVantage returns a key that includes the selected interval, e.g. “Time Series (60min)”, “Time Series (30 min)” etc., but the general idea is to have something like this in the module’s communication section:

{
   ...
   "response": {
      "output": {
         "timeSeries": "{{toArray(body.someCollection)}}"
      }
   }
   ...
}

Thanks for getting me pointed in the right direction!

John

2 Likes