Google Vertex AI (Gemini) - Analyze Image/Video (gemini-pro-vision)

[Samuel1]

Hi Makers,

We are trying to create an automation process that scrapes all the relevant data from competitors’ Instagram accounts, including media URLs, like counts, comment counts, and Instagram permalinks. Additionally, we would like to obtain descriptions of the posts. We are able to achieve this for images using OpenAI’s Analyze Image feature, but it does not work for videos. To address this issue, we explored different options and found Google Vertex AI (Gemini), which offers Image/Video Analysis.

We have set up every possible step for the module and tried all the options in the regions, roles, file upload types, MIME types, and file data, but nothing seems to work. We frequently encounter “400” errors, indicating that the request contains an invalid argument.

I attempted to reach out to support, but it is not possible to select Google Vertex AI (Gemini) as the name of the app when submitting a support request.

Does anyone have experience setting up Google Vertex AI in general? I have a feeling that the Google Vertex App either does not work at all or requires very complex setup steps.

[dcturner1]

Curious if you got an answer to this? Looking for a similar solution

[Samuel1]

Hi,

I did not, but I found a solution . You need to add the HTTP module (Get a file) with the media URL from Instagram before Gemini. The setup in Gemini could look like this:

[vendy]

Hello there @Samuel1

Great to hear that you’ve found a solution to your struggles!

We appreciate you sharing the solution with us and maintaining our community organized.

[Adrien_C]

Hi Samuel,

Thanks for the solution.
Is Gemini giving you good responses?

Because I tried analyzing some videos and Gemini always hallucinates; the answers are very far from the content of my videos.

Perhaps I missed something in the settings…

[Samuel1]

Hey Adrien_C,

For some videos, it gives good responses, but for others, it hallucinates quite a lot (once I even received an answer with Chinese characters ).

In my opinion, it is more on the side of Gemini than the settings.

[Daniel_Alisov]

Hey makers! i also tried this setup - it always write this

so eventually i created some comlex bot in telegram that i can can just paste a link of instagram reels or youtube shorts and he will download the media using apify actors then analyze it using open-ai or anthrophic and then return data bacl to me based assistant that know my style in social and adapt the video i sent as link and create a post text or video scene for me