Google Vertex AI (Gemini) - Metadata Tokens not showing up

How To
, ,
1

I am using the Vertex AI to run a messages chat to get a response.
Everything is working fine but I want to check how many tokens were used and store them in a google sheet, but the data which is showing up for tokens is not available when I am either trying to compose a string or update directly

The tokens show up in the response

image
image425Ă—497 26.4 KB

But when I am trying to put that as a string

image
image637Ă—427 32.4 KB

I get no inputs as if the value is not there

image
image439Ă—358 23.4 KB

Thanks in advance for all the help

2

Can you expand the output so we can see what you have there?

L

3

Hi thanks for the reply.

Here is the output from the vertex Gemini API

image
image385Ă—850 27.9 KB

4

Hmm, I never saw this reply.

First, did you resolve this?

If not…

Do all the collections in the response have the same structure? Or only the last prediction? If it’s the last prediction, then you might need to use the index of the last prediction to get the tokens count

If not, please share your input and output bundles. You can remove or mal sensitive output/input if needed. But the more complete the info, the easier it will be to help you.

L

5

Hey I wasn’t able to, I was working on other parts of the problem in the flow

but here is the input and output bundle - btw really appreciate the help
Input -
[
{
“topK”: 15,
“topP”: 1,
“model”: “gemini-1.5-pro”,
“messages”: [
{
“role”: “user”,
“content”: “” },
{
“role”: “model”,
“content”: “Sure I can help you with that.”
},
{
“role”: “user”,
“content”: “”
}
],
“projectId”: “betterwithaihackathon”,
“temperature”: 0.9,
“maxOutputTokens”: 5000,
“serviceEndpointLocationId”: “europe-west9”
}
]

Output -
[
{
“textResponse”: “”,
“predictions”: [
{
“candidates”: [
{
“content”: {
“role”: “model”,
“parts”: [
{
“text”: “{”
}
]
}
}
]
},
{
“candidates”: [
{
“content”: {
“role”: “model”,
“parts”: [
{
“text”: “\n "headline": "”
}
]
},
“safetyRatings”: [
{
“category”: “HARM_CATEGORY_HATE_SPEECH”,
“probability”: “NEGLIGIBLE”,
“probabilityScore”: 0.15546274,
“severity”: “HARM_SEVERITY_NEGLIGIBLE”,
“severityScore”: 0.13106197
},
{
“category”: “HARM_CATEGORY_DANGEROUS_CONTENT”,
“probability”: “NEGLIGIBLE”,
“probabilityScore”: 0.15687835,
“severity”: “HARM_SEVERITY_NEGLIGIBLE”,
“severityScore”: 0.049958523
},
{
“category”: “HARM_CATEGORY_HARASSMENT”,
“probability”: “NEGLIGIBLE”,
“probabilityScore”: 0.17384852,
“severity”: “HARM_SEVERITY_NEGLIGIBLE”,
“severityScore”: 0.1046602
},
{
“category”: “HARM_CATEGORY_SEXUALLY_EXPLICIT”,
“probability”: “NEGLIGIBLE”,
“probabilityScore”: 0.28866935,
“severity”: “HARM_SEVERITY_NEGLIGIBLE”,
“severityScore”: 0.12940271
}
]
}
]
},
{
“candidates”: [
{
“content”: {
“role”: “model”,
“parts”: [
{
“text”: “”
}
]
},
“finishReason”: “STOP”
}
],
“usageMetadata”: {
“promptTokenCount”: 1311,
“candidatesTokenCount”: 299,
“totalTokenCount”: 1610
}
}
]
}
]

I have truncated the output bundle there are different number of candidates every time under predictions

6

Here is a bundle that takes the output you gave me and generates the number of tokens used.

The important part is that it’s supposed to adapt to the length of the incoming input (the use of the length() function does that).

blueprint (4).json (8.3 KB)

Hopefully that gives you some ideas.

L

7

Thanks a lot will try ( I honestly just stopped using Gemini and moved to other options)

Will mostly be using something like hugging face apis or something simmilar in future.

Thanks

8

Hello!

I also had this problem. Here is the solution:

{{last(map(XX.predictions; “usageMetadata.promptTokenCount”))}}

{{last(map(XX.predictions; “usageMetadata.candidatesTokenCount”))}}

{{last(map(XX.predictions; “usageMetadata.totalTokenCount”))}}

Best Regards,

1 Like