Hello all! very new to this, but I am trying to have assembly AI make a transcript of recordings roughly 45min- 1 hour long and then send that transcript to chat gpt 4 for analysis. My test run was about 15k tokens which hits above the limit. (I know a 3.5 turbo 16k exists but thats not optimal). Chat gpt suggested breaking into chunks but could not figure out the codes to do so and eventually told me to move onto something else lmao. Problem is I can break the transcript into 2, but I dont want 2 different results, but I want both halves or chunks to eventually be combined and have the result / analysis based off the full transcript. Anyone know how to go about this??? Thanks.
Welcome to the Make Community!
You’ll have to split your input into chunks below the maximum token context window size.
You can estimate the number of tokens in text, as well as split them up into multiple bundles using my modules found in my latest forum topic found on my profile page.
Hope this helps! Let me know if there are any further questions or issues.
— @samliew