Skip to content

fix(models): keep url_context retrieval status on model responses - #8236

Open
karthik-0306 wants to merge 1 commit into
google:mainfrom
karthik-0306:fix/8226-keep-url-context-metadata
Open

karthik-0306 wants to merge 1 commit into
google:mainfrom
karthik-0306:fix/8226-keep-url-context-metadata

Conversation

@karthik-0306

Copy link
Copy Markdown

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

Problem:
Gemini reports which URLs the url_context tool retrieved, and the status of each retrieval (SUCCESS, ERROR, PAYWALL, UNSAFE), in candidate.url_context_metadata. LlmResponse had no field for it, so both LlmResponse.create and the SSE StreamingResponseAggregator dropped it. To callbacks, plugins and stored events, a failed or paywalled retrieval looked the same as a successful one.

Solution:

  • Add url_context_metadata: types.UrlContextMetadata | None to LlmResponse.
  • Set it in both the success and error branches of LlmResponse.create.
  • Carry it through every path of StreamingResponseAggregator (progressive and non-progressive), the same way citation_metadata is.

The flow finalizer already copies every LlmResponse field onto the Event, so no flow change is needed. The v1 session schema stores events as JSON, so the field is persisted without a migration. The legacy v0 layout reports it through the existing warn_event_fields_not_stored mechanism, like other fields added after that layout froze.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

All four new cases fail on main and pass with this change:

  • tests/unittests/models/test_llm_response.py: test_llm_response_create_keeps_url_context_metadata[success] and [error].
  • tests/unittests/utils/test_streaming_utils.py: url_context_default and url_context_progressive cases added to the existing parametrized test_close_preserves_metadata. The metadata arrives only on the first chunk, so the test checks that aggregation keeps it.

pytest results:

  • Both files: 109 passed.
  • sessions/test_session_util.py, sessions/test_v0_storage_event.py, events/, models/test_llm_response.py, utils/test_streaming_utils.py: 394 passed.
  • Broader suites (sessions, a2a, cli, plugins): identical to main in my environment.
  • ruff, isort, pyink, codespell and mypy are clean on the changed files.

Manual End-to-End (E2E) Tests:

Ran an InMemoryRunner with a Gemini model (only api_client mocked) and the url_context tool, where the model's response reports URL_RETRIEVAL_STATUS_ERROR:

  • After: the stored session event has url_context_metadata with URL_RETRIEVAL_STATUS_ERROR in both StreamingMode.NONE and StreamingMode.SSE.
  • Before: on main the same script raises AttributeError: 'Event' object has no attribute 'url_context_metadata'.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Additional context

This adds a public field to LlmResponse / Event. EXCLUDED_EVENT_FIELDS in the agent test runner is unchanged: the field is deterministic grounding data like grounding_metadata, and the event dumps there use exclude_none, so existing test fixtures are unaffected. I'm happy to open a follow-up docs PR if you'd like the field documented.

Gemini reports which URLs the url_context tool retrieved, and whether each
retrieval succeeded, in candidate.url_context_metadata. LlmResponse had no
field for it, so LlmResponse.create and the streaming aggregator dropped it,
and a failed or paywalled retrieval looked the same as a successful one to
callbacks, plugins and stored events. Add the field, set it in both branches
of LlmResponse.create, and carry it through the streaming aggregator the way
citation_metadata is.

Fixes google#8226
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

url_context tool's retrieval status is discarded: LlmResponse drops candidate.url_context_metadata

2 participants