Skip to content

ci: release workflow test gate is dead — continue-on-error forces conclusion=success; flip to outcome + make CI suite honestly green #255

Description

@rahlk

PROBLEM

The release workflow's test gate is dead code and has never fired. .github/workflows/release.yml runs tests with continue-on-error: true (line 44) and gates tag deletion on steps.test.conclusion == 'failure' (line 48) — but under continue-on-error a failed step's conclusion is forced to success; only outcome records the failure. Verified on real runs: v1.4.0 and v1.4.1 both had failing tests in Run Tests, conclusion=success, Delete-tag skipped, release published.

SCOPE BOUNDARY

The release workflow's gate, plus the suite hygiene needed to make flipping it safe. Not PR-time CI, which is its own concern. Do not just flip the field — see the caveat.

GOALS

  • steps.test.conclusionsteps.test.outcome, so the gate is real. (Landed on release/2.0 via Fix dead release test gate: use outcome, not conclusion #306.)
  • Partition the CI suite so "green" is meaningful: env-dependent tests skip cleanly via pytest markers or fixture-presence gating, rather than failing.
  • Companion failures resolved — TS golden-fixture collection errors, Java source-code-path crash.
  • Same fix applied to main, not only release/2.0.

CAVEATS AND KNOWN RISKS

  • Flipping the field alone blocks every release. The gate becomes real against a suite that is not currently CI-green, so the first tag after the flip fails and stays failing. That is why the flip and the suite work are one issue and not two.
  • continue-on-error must stay. Removing it fails the job immediately and the tag-deletion step never runs — the opposite of the intent. The two settings are a pair, and Fix dead release test gate: use outcome, not conclusion #306 comments them as such.
  • A gate that fires deletes the pushed tag. Worth exercising on a scratch tag before trusting it on a real release.

DEFINITION OF DONE

  • Gate fires on a deliberately broken test, verified on a scratch tag or a workflow_dispatch dry run — demonstrated, not reasoned about.
  • Release workflow green on an honest, fully-run suite; tests skipped by gating are visibly reported rather than silently absent.
  • Both release lines carry the fix.

Metadata

Metadata

Assignees

No one assigned

    Labels

    cicontinuous integration goodies

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions