fix(docker): preserve single-url crawl failure details

[nightcityblade]

Summary

Fixes #2116

Treat unsuccessful crawls from the single-URL /md and /llm/{url} handlers as upstream failures (502 Bad Gateway) so the existing exception handler preserves error_message instead of replacing it with an opaque internal-server-error response. The LLM handler now also preserves deliberate HTTPException statuses.

List of files changed and why

• deploy/docker/api.py - centralize unsuccessful crawl handling and return a detail-preserving 502 from both single-URL handlers.
• deploy/docker/tests/test_api_crawl_failures.py - verify the 502 mapping, preserved anti-bot reason, both handler call sites, and LLM exception propagation.

How Has This Been Tested?

• .venv/bin/python -m pytest deploy/docker/tests/test_api_crawl_failures.py -q (3 passed)
• .venv/bin/python -m pytest deploy/docker/tests/test_security_headers_xss.py::TestErrorSanitization -q (2 passed)
• .venv/bin/python -m ruff check --select E9,F63,F7,F82 deploy/docker/api.py deploy/docker/tests/test_api_crawl_failures.py (passed)
• Direct exception-handler smoke check confirmed a 502 response retains the anti-bot detail.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas (N/A: no complex logic was introduced)
• I have made corresponding changes to the documentation (N/A: no configuration or usage change)
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[nightcityblade]

CI note: the code-related posture-gate and security-offline checks both pass. The only failing job is the Discord notification workflow; its log shows an empty DISCORD_WEBHOOK and exits before exercising this change. This is repository/fork-secret configuration rather than a patch failure.

[bong-u]

Nice work.
I hit the same 500 from a client and found one more spot the issue didn't cover.

deploy/docker/server.py:911:

if all(not result["success"] for result in results["results"]):
    raise HTTPException(500, f"Crawl request failed: {results['results'][0]['error_message']}")

For a single-URL request "all failed" is just "it failed", so the detail gets logged and dropped by the same 500 branch. Stock 0.9.2:

POST /crawl {"urls":["https://www.sagaftra.org/"]}
→ HTTP 500
→ {"error":"Internal server error","correlation_id":"844da2acf429"}

That's the same body the issue reports for /md, from the endpoint it lists as unaffected.
server.py:911 is byte-identical on v0.9.2, main and this PR's head.

The detail is already interpolated here, so switching that 500 to 502 is enough to make it survive.
If you'd rather reuse _raise_for_crawl_failure, it won't drop in as-is: handle_crawl_request returns model_dump() dicts (api.py:751), so it's result["success"].

[nightcityblade]

Thanks for the concrete reproduction. You're right: the all-results-failed branch in /crawl also raises 500, so the issue's statement that /crawl is unaffected is too broad. This PR is scoped to the two single-URL handlers from #2116, and I don't want to expand it without maintainer direction. I'm happy to include that adjacent branch here if the maintainers prefer, or it can be tracked separately.

[bong-u]

That makes sense.
My main goal was to make sure the /crawl case doesn't get overlooked.
I agree that maintainer preference should decide whether it belongs here or in a follow-up.
Thanks for confirming the behavior.