Skip to content

docs: explain GITHUB_TOKEN forwarding for eval containers - #3857

Merged
Sayt-0 merged 1 commit into
docker:mainfrom
priyanka25aug:docs/eval-github-token-forwarding
Jul 28, 2026
Merged

docs: explain GITHUB_TOKEN forwarding for eval containers#3857
Sayt-0 merged 1 commit into
docker:mainfrom
priyanka25aug:docs/eval-github-token-forwarding

Conversation

@priyanka25aug

Copy link
Copy Markdown
Contributor

Fixes #3841

Eval containers deliberately do not receive GITHUB_TOKEN / GH_TOKEN, even though dedicated provider API keys are forwarded automatically (nonForwardableTokenEnvVars in pkg/config/auto.go excludes GITHUB_TOKEN). This is intentional but undocumented, and the asymmetry with other providers is confusing.

This PR adds a warning to the evaluation docs and the GitHub Copilot provider docs:

  • How to pass the token explicitly with -e GITHUB_TOKEN
  • That --env-from-file alone is not sufficient
  • That the LLM judge runs on the host, so judge scoring can succeed while the evaluated agent fails to authenticate

@priyanka25aug
priyanka25aug requested a review from a team as a code owner July 27, 2026 23:14
@aheritier aheritier added area/docs Documentation changes kind/docs Documentation-only changes labels Jul 27, 2026

@Sayt-0 Sayt-0 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good thank you for the contribution !

@Sayt-0

Sayt-0 commented Jul 28, 2026

Copy link
Copy Markdown
Member

One more thing : can you sign your commits ? It's mandatory in this repo

@priyanka25aug
priyanka25aug force-pushed the docs/eval-github-token-forwarding branch from c17d62e to 76c1051 Compare July 28, 2026 13:48
GITHUB_TOKEN and GH_TOKEN are not forwarded into eval containers by design
(nonForwardableTokenEnvVars in pkg/config/auto.go). Add a WARNING callout
with the -e GITHUB_TOKEN workaround to both the evaluation docs and the
GitHub Copilot provider docs.

Fixes docker#3841

Signed-off-by: Priyanka Bajaj <[email protected]>
@priyanka25aug
priyanka25aug force-pushed the docs/eval-github-token-forwarding branch from 76c1051 to badc210 Compare July 28, 2026 13:58
@Sayt-0
Sayt-0 merged commit 877df71 into docker:main Jul 28, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/docs Documentation changes kind/docs Documentation-only changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Docs: Explain GITHUB_TOKEN forwarding for eval

4 participants