What

LocalTokenizer.compute_tokens() builds roles with one entry per Part:

for content in processed_contents:
  text_accumulator.add_content(content)
  if content.parts:
    for _ in content.parts:
      roles.append(content.role)

but _TextsAccumulator does not emit one text per part. A function_call / function_response part contributes the function name plus every key and every string value of its args/response as separate texts; a thought_signature-only part contributes none. The two lists are then combined with zip() at both tokenizer branches (lines 386 and 405), and zip silently truncates to the shorter one.

Effect

Model turn with function_call(get_weather, {location: "NYC"}), then a user text:

texts actually sent to the tokenizer:  'get_weather', 'location', 'NYC', "What's the weather?"
TokensInfo entries returned:           2
  role='model'  tokens=['get_weather']
  role='user'   tokens=['location']        <- model's own argument, labelled user

Two of the four texts are dropped from the result, and the surviving entries carry the wrong role. With a thought_signature-only part first (routine on Gemini 2.5/3 function-calling and thinking turns), the user's own text is labelled role='model'.

TokensInfo.role is documented as "the role from the corresponding Content" (types.py), which this violates. The output also matches neither "one entry per part" nor "one entry per text", so it isn't a defensible alternate semantics.

Reproduced on the real gemma3 SentencePiece model (_local_tokenizer_loader's pinned download) and, for the HuggingFace/gemma4 branch, with the same mock pattern test_local_tokenizer.py already uses for that branch.

Fix

Extend roles by the number of texts the accumulator actually added for each Content:

roles: list[Optional[str]] = []
...
for content in processed_contents:
  texts_before = len(text_accumulator)
  text_accumulator.add_content(content)
  roles.extend([content.role] * (len(text_accumulator) - texts_before))

with a two-line _TextsAccumulator.__len__. Both zip() sites are now aligned with get_texts() by construction. Plain-text contents add exactly one text per part, so their behaviour is unchanged.

After the fix the example above returns 4 entries with roles model, model, model, user.

Verification

• pytest google/genai/tests/local_tokenizer/: 35 passed (33 existing + 2 new). The 2 new tests fail on main.
• mypy google/genai/local_tokenizer.py with the repo's strict config: 0 errors in this file, before and after. (My environment reports 7 pre-existing errors in _api_client.py / _local_tokenizer_loader.py / errors.py that are identical with and without this change — dependency-version noise, not from this diff.)

+12 −4 in local_tokenizer.py, plus one test file covering both branches.

CLA

I'll complete the Google CLA when the bot prompts.

Investigated and fixed with AI assistance (Claude); I reviewed the change and ran the verification above myself.