Dynamically Pass an Array of Images in JSON to Claude or OpenAI Vision

[Manita112]

I have a problem to pass dynamicly multiple images to the message of Claude.
I want to make a description based on the product images and Knowledge base.

My Blueprint:
blueprint (2).json (73.8 KB)

What do i need to change? or is this only possible with a API Request?
I run in the same issue with ChatGPT-Vision, when i have tried to map multiple images with JSON format.

I have tried to build the JSON body like in “messages” how Anthropic descript in the documention:

Vision - Anthropic

message = client.messages.create(
    model="claude-3-opus-20240229",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Image 1:"
                },
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": image1_media_type,
                        "data": image1_data,
                    },
                },
                {
                    "type": "text",
                    "text": "Image 2:"
                },
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": image2_media_type,
                        "data": image2_data,
                    },
                },
                {
                    "type": "text",
                    "text": "How are these images different?"
                }
            ],
        }
    ],
)

This is my Message in 19:
BASE64(DATA) is current only a variable to show how i parse the base64 in the step before, the content is too much to post here.

[
   {
      "role":"user",
      "content":[
         {
            "type":"text",
            "text":"Image 1:"
         },
         {
            "type":"image",
            "source":{
               "type":"base64",
               "media_type":"image/jpeg",
               "data":"BASE64(DATA)"
            }
         },
         {
            "type":"text",
            "text":"Image 2:"
         },
         {
            "type":"image",
            "source":{
               "type":"base64",
               "media_type":"image/jpeg",
               "data":"BASE64(DATA)"
            }
         },
         {
            "type":"text",
            "text":"Image 3:"
         },
         {
            "type":"image",
            "source":{
               "type":"base64",
               "media_type":"image/jpeg",
               "data":"BASE64(DATA)"
            }
         }
      ]
   }
]

[TheWolfyAmongUs]

Sorry haven’t tried it out myself, but it looks like your JSON is missing a comma after the ending bracket for “source”.
I’ve seen your blueprint, and wanted to know if you can use the Create JSON module instead of concatenating text yourself?

[Manita112]

The generated json ist valid, like i tested it online. I cant debug it, because i dont see the request Make does to Claude.

I have tried it with the JSON Modul, but my research hit me to do it this way.

Combine Bundle In 1 Json

Use a text aggregator first: [Screenshot_2023-10-20_091010] Then in the module where you need to map the JSON, add the opening and closing brackets: [Screenshot_2023-10-20_091011]

because i need to generate it dynamic, based on the amount of the uploaded images.

Is it possible that i cant map value like a API call?

[Manita112]

I have figured out the structure and need to convert into a array in the last step, now i have a issue to pass the image through in the right format to the request.

[
    {
        "array": [
            {
                "role": "user",
                "content": [
                    {
                        "type": "image",
                        "source": {
                            "filename": "file.png",
                            "data": "DATA"
                        }
                    }
                ]
            }
        ],
        "__IMTAGGLENGTH__": 1
    }
]

Current Error:
“The operation failed with an error. [400] messages.0.content.0.image.source.base64.data: The image was specified using the image/png media type, but does not appear to be a valid png image ”

Now i need to figure out, how i can pass the images through the request.
If i dont wrap the data 54 in base64(), i can decode the image online correctly, after the wrapping its not possible.

I tried it to with base64(), toBinary(;base64), without converting

Have somebody a idea to fix this issue?

Thanks
Jamie

[techpeace]

This turned out to hold the solution for me:

Value not found in options in parameter 'imageUploadType'

Maybe instead of Create and Parse JSON in your Iterator-Aggregator loop, You can try setting the Target Structure field of your aggregator to the Images field (it might or might not work sometimes). The Array Aggregator module also allows you to build a complex array of collections for a later module’s field to map multiple items (collections) to it. Here is an example of using the “Target structure type” of an Array Aggregator module: [Screenshot_2024-01-17_100113] As you can see, the “Ma…

You can set the target structure for your aggregator module to the Images field for OpenAI:

Then in the OpenAI module, select Map, and choose the output of your aggregator module:

Unfortunately, that approach doesn’t work for Anthropic, due to the structure of the request body. I’ll report back in this thread if I find an approach that works.