Race condition in `CredentialManager.get_auth_credential()` — missing concurrency control in credential exchange/refresh/save lifecycle

[liuchunyi-buaa]

🔴 Required Information

Title: Race condition in CredentialManager.get_auth_credential() — missing concurrency control in credential exchange/refresh/save lifecycle

Describe the Bug

CredentialManager.get_auth_credential() orchestrates a multi-step credential lifecycle (load → exchange → refresh → save) with no concurrency control. When multiple async coroutines invoke this method concurrently for the same AuthConfig, each independently executes the full lifecycle, causing:

1. Duplicate token exchange: Multiple callers independently hit the token endpoint
2. Refresh token invalidation: One coroutine refreshes, then another refreshes again, potentially invalidating the first refresh token (many OAuth2 providers require single-use refresh tokens)
3. Credential save race: Multiple coroutines race to call _save_credential(), potentially leaving incomplete or corrupted state in the credential service

The class-level _registry_lock only protects provider/exchanger/refresher registration, not credential operations.

Steps to Reproduce

1. Install the google-adk package
2. Configure an Agent using OAuth2 authentication (with issuer_url, client_id, client_secret)
3. Trigger concurrent get_auth_credential() calls:

import asyncio
from google.adk.auth.credential_manager import CredentialManager

async def test_race_condition():
    # Construct an AuthConfig with OAuth2 scheme + raw credential
    config = ...

    # Multiple coroutines concurrently request the same credential
    results = await asyncio.gather(*[
        CredentialManager(config).get_auth_credential(ctx)
        for _ in range(5)
    ])

    # Observed: each coroutine independently completes exchange → refresh → save
    # Expected: only one coroutine should process; others should await the result

4. Inspect logs for duplicate token exchange/refresh calls
5. Verify whether the credential service contains consistent final state

Expected Behavior

The credential lifecycle should execute atomically under a per-credential-key asyncio.Lock. Concurrent callers should follow the double-checked locking pattern: re-verify whether the credential has already been processed by another caller after acquiring the lock.

Observed Behavior

# credential_manager.py:180-274 — the core flow, entirely lock-free

async def get_auth_credential(self, context) -> Optional[AuthCredential]:
    # Step 2: Check if credential is already ready
    raw_auth_credential = self._auth_config.raw_auth_credential
    if self._is_credential_ready() and raw_auth_credential is not None:
        return raw_auth_credential.model_copy(deep=True)

    # Step 3: Load existing credential          ← no lock
    credential = await self._load_existing_credential(context)

    # Step 6: Exchange credential if needed     ← no lock
    credential, was_exchanged = await self._exchange_credential(credential)

    # Step 7: Refresh credential if expired     ← no lock
    was_refreshed = False
    if not was_exchanged:
        credential, was_refreshed = await self._refresh_credential(credential)

    # Step 8: Save credential if modified       ← no lock
    if was_from_auth_response or was_exchanged or was_refreshed:
        await self._save_credential(context, credential)

    return credential

No mutual exclusion guards any of these steps. Five concurrent coroutines at Step 3 can all read the same expired token, then each independently refreshes it at Step 7.

Environment Details

• ADK Library Version (pip show google-adk): audited code snapshot (commit: 3a5c6b33)
• Desktop OS: Windows 11
• Python Version (python -V): 3.13

Model Information

• Are you using LiteLLM: N/A (this issue does not involve model calls)
• Which model is being used: N/A

---

🟡 Optional Information

Regression

N/A (present since initial design; not a regression)

Minimal Reproduction Code

"""Minimal example reproducing the CredentialManager race condition"""
import asyncio

async def demo_race_condition():
    # Simulate: 5 concurrent callers each independently
    # walk the full exchange → refresh → save lifecycle
    # starting from the same stale token

    async def credential_lifecycle(name: str):
        # Step 3: all coroutines read the same expired token
        token_before = {"access_token": "expired_xxx", "refresh_token": "rt_yyy"}

        # Step 6-7: each independently refreshes
        # (in practice, only one should execute)
        print(f"[{name}] Refreshing token...")
        await asyncio.sleep(0.1)  # simulate network I/O
        token_after = {"access_token": f"new_token_from_{name}"}

        # Step 8: each independently saves
        print(f"[{name}] Saving credential...")
        return token_after

    results = await asyncio.gather(*[
        credential_lifecycle(f"coroutine_{i}") for i in range(5)
    ])

    # The last writer overwrites all prior results
    print(f"\nProcessed {len(results)} times (expected: 1)")
    for r in results:
        print(f"  {r}")

asyncio.run(demo_race_condition())

Sample output:

[coroutine_0] Refreshing token...
[coroutine_1] Refreshing token...
[coroutine_2] Refreshing token...
[coroutine_3] Refreshing token...
[coroutine_4] Refreshing token...
[coroutine_0] Saving credential...
[coroutine_1] Saving credential...
[coroutine_2] Saving credential...
[coroutine_3] Saving credential...
[coroutine_4] Saving credential...

Processed 5 times (expected: 1)

How often has this issue occurred?

• Always (100%) — every concurrent get_auth_credential() invocation on the same AuthConfig triggers it

Suggested Fix

import asyncio

class CredentialManager:
    def __init__(self, auth_config, ...):
        ...
        self._credential_locks: dict[str, asyncio.Lock] = {}

    async def get_auth_credential(self, context) -> Optional[AuthCredential]:
        key = self._auth_config._credential_key()
        lock = self._credential_locks.setdefault(key, asyncio.Lock())

        async with lock:
            # Double-check: re-verify whether another coroutine already
            # processed the credential while we were waiting for the lock
            if self._is_credential_ready():
                return self._auth_config.raw_auth_credential.model_copy(deep=True)

            credential = await self._load_existing_credential(context)
            # ... existing exchange / refresh / save logic ...

Additional Context

ADK is designed for multi-agent systems where multiple agents may share OAuth2 credential configurations. Under high-concurrency agent orchestration, this race condition is likely to be triggered frequently. While the primary impact is reliability (token invalidation, duplicate API calls) rather than security (no privilege escalation), atomicity in the authentication subsystem should be treated as a correctness requirement.

[tylergibbs1]

I would like to work on this. I will first add a regression test that runs concurrent credential requests against the same credential key, then add narrowly scoped per-key async locking with a second readiness check after lock acquisition. The implementation will preserve concurrency for different credentials and avoid holding a global lock across network I/O. Please let me know if this direction conflicts with any planned auth changes.

[surajksharma07]

@liuchunyi-buaa Confirmed the race on our end — walked through get_auth_credential() and reproduced it: with no lock, 5 concurrent callers each redo the full exchange/refresh/save (5/5), exactly as reported.

PR #6478's per-key lock with a reload-after-acquire check lines up with what we tried too, and got that down to 1/1 in our own testing.

@tylergibbs1 could you double-check the lock is scoped at the class level rather than per-instance?

Each concurrent caller here constructs its own CredentialManager, so a per-instance lock wouldn't help.
Worth confirming that holds under the full test suite before this gets merged!