What are you trying to achieve?
I’m trying to make a system where I can get info from an API and search through it for specific data points, and then run the selected data through Chat GPT to analyse it, so basically a search engine for a massive amount of data from an API. Lets say I have info on 1000 companies, so far I’ve managed to pull the info, I’ve got it all into one long text (takes a while to load when opening), then parsed it so there are 1000 files (takes even longer), but I don’t know how to put it through Chat GPT as such a large data set. How can I filter or search through the data so I can only access the relevant data, and only put that* data through a chat gpt prompt. I’ve been practicing trying to look for the company Apple but I don’t really know how to approach this. Eventually I want to have info on all companies referenced by the API, which is pretty much the entire stock market, so I figured 1000 companies is a good place to start. I know it’s possible to do quickly because google exists. What modules should I use in what order? And how do I limit loading times? Help please !
Steps taken so far
Pulled the 1000 companies data from the API, parsed them so they are 1000 files, ran into mad problems trying to basically do anything further than that, I’ve messed around with aggregators and iterators trying to address a different problem that I won’t go into here, but I thought I’d just take a step back and ask if there are any better ways to approach this that I’m not seeing.