refactor(amber): read the loop input table from its materialization instead of shipping it in state

[aglinxinyuan]

What changes were proposed in this PR?

The loop's input table used to ride inside the State content: LoopStart encoded its buffered input as Arrow IPC bytes, base64'd into the JSON content column, and that payload was re-written and re-read at every loop-body hop, every iteration.

That data already exists. In the fully-materialized mode loops require, the Loop Start's input-port materialization holds exactly the loop's input table for the whole loop — Loop Start re-reads it every iteration, and the back-edge truncates only the state doc at the same base URI, never the result doc. So this PR ships the port's base URI in the setup config and derives both addresses from it:

loopStartPortUris[LoopStart-id] = <base URI of LoopStart's input port>
        ├── state_uri(base)   → back-edge write address   (as before, derived)
        └── result_uri(base)  → the loop's input table    (NEW: read at consume)

Piece	Before	After
proto field 4	loopStartStateUris = state URI	loopStartPortUris = base URI (renamed so the semantic change is loud)
LoopStart's produced state	user vars + IPC-encoded table (base64 in JSON)	user vars only — small, pure JSON
Loop-body hops	fat state re-materialized per hop, per iteration	tiny state
LoopEnd's table	decoded from state content	read once per iteration from result_uri(base), injected via a new runtime-only attach_loop_table hook; a read failure is reported like a UDF error
table_to_ipc_bytes / table_from_ipc_bytes	second, divergent Arrow codec (lossy from_pandas inference)	deleted — the read goes through the canonical iceberg reader

Semantics deliberately preserved:

• the reserved-table collision raise stays (a user var named table would now be silently shadowed by the injected table — worse than before);
• the "consumed" marker (_loop_table) is still set only by a successful run_update, so condition()'s short-circuit for pass-through-only Loop Ends is unchanged;
• nested loops work by construction: the inner Loop Start's entry points at the outer Loop Start's output port, whose result doc is recreated per outer iteration but persists across inner iterations (the jump rewinds to the inner level only).

Wins: no ~33% base64 bloat, no JSON-column size ceiling on the table (large-table loops become viable), strictly less I/O for any non-empty loop body (one read per iteration replaces N state-doc writes+reads per hop), and one Arrow codec instead of two.

Note: this deepens the read-side use of storagePairs.head._1 — the same shared upstream URI as the known back-edge fan-out design discussion; if that ever moves to a per-loop private doc, this read moves with it.

Any related issues, documentation, discussions?

Builds on #5900 (State columns) and #6661 (envelope through JVM hops). Related design context: #6660.

How was this PR tested?

• Unit — test_loop_operators.py rewritten for the attach-based flow plus new pins: the produced state carries no table; attaching alone does not mark the loop consumed; run_update fails loud when no table was attached. test_main_loop.py pins the base-URI derivation for the back-edge write (state_uri(base)), the missing-config fail-loud, and that the matching consume injects the read table. test_initialize_executor_handler.py covers the renamed proto field. 237 tests green locally (the only failures in a full sweep are pre-existing environment ones, identical on unmodified main).
• Scala — full test-compile (proto regen included), scalafmtCheckAll, scalafixAll --check, and the worker/descriptor spec suites (WorkerSpec, WorkflowWorkerSpec, SerializationManagerSpec, WorkflowExecutionManagerSpec, LoopStartOpDescSpec, LoopEndOpDescSpec) all pass on Java 17.
• E2E — the four LoopIntegrationSpec cases (single, nested 3×3, JVM chain, nested JVM chain) exercise the full read-at-consume path in the amber-integration CI job; the nested cases specifically cover the inner-loop read against the outer Loop Start's per-outer-iteration output doc.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Fable 5)

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• Contributors with relevant context: @Yicong-Huang
  You can notify them by mentioning @Yicong-Huang in a comment.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 75.86207% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.97%. Comparing base (5040cad) to head (9204ed1).

Files with missing lines	Patch %	Lines
amber/src/main/python/core/runnables/main_loop.py	56.25%	7 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #6971      +/-   ##
============================================
- Coverage     78.97%   78.97%   -0.01%     
- Complexity     3783     3787       +4     
============================================
  Files          1160     1160              
  Lines         46105    46110       +5     
  Branches       5115     5115              
============================================
+ Hits          36413    36415       +2     
- Misses         8068     8075       +7     
+ Partials       1624     1620       -4     

Flag	Coverage Δ		*Carryforward flag
access-control-service	70.00% <ø> (ø)
agent-service	76.76% <ø> (ø)		Carriedforward from 5040cad
amber	72.19% <100.00%> (+0.02%)	⬆️
computing-unit-managing-service	20.49% <ø> (ø)
config-service	66.66% <ø> (ø)
file-service	67.21% <ø> (ø)
frontend	82.99% <ø> (ø)		Carriedforward from 5040cad
notebook-migration-service	78.94% <ø> (ø)
pyamber	95.23% <74.07%> (-0.15%)	⬇️
workflow-compiling-service	26.31% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 9 worse · ⚪ 6 noise (<±5%) · 0 without baseline

Compared against main 5040cad benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	353	0.215	27,596/38,666/38,666 us	🔴 +22.4% / 🔴 +138.2%
🔴	bs=100 sw=10 sl=64	758	0.463	124,915/192,394/192,394 us	🔴 +34.1% / 🔴 +77.1%
⚪	bs=1000 sw=10 sl=64	899	0.548	1,115,167/1,140,207/1,140,207 us	⚪ within ±5% / 🔴 +11.6%

Baseline details

Latest main 5040cad from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	353 tuples/sec	410 tuples/sec	767.9 tuples/sec	-13.9%	-54.0%
bs=10 sw=10 sl=64	MB/s	0.215 MB/s	0.25 MB/s	0.469 MB/s	-14.0%	-54.1%
bs=10 sw=10 sl=64	p50	27,596 us	22,554 us	12,502 us	+22.4%	+120.7%
bs=10 sw=10 sl=64	p95	38,666 us	35,496 us	16,234 us	+8.9%	+138.2%
bs=10 sw=10 sl=64	p99	38,666 us	35,496 us	18,919 us	+8.9%	+104.4%
bs=100 sw=10 sl=64	throughput	758 tuples/sec	815 tuples/sec	974.8 tuples/sec	-7.0%	-22.2%
bs=100 sw=10 sl=64	MB/s	0.463 MB/s	0.497 MB/s	0.595 MB/s	-6.8%	-22.2%
bs=100 sw=10 sl=64	p50	124,915 us	120,883 us	102,449 us	+3.3%	+21.9%
bs=100 sw=10 sl=64	p95	192,394 us	143,460 us	108,652 us	+34.1%	+77.1%
bs=100 sw=10 sl=64	p99	192,394 us	143,460 us	116,310 us	+34.1%	+65.4%
bs=1000 sw=10 sl=64	throughput	899 tuples/sec	895 tuples/sec	1,004 tuples/sec	+0.4%	-10.5%
bs=1000 sw=10 sl=64	MB/s	0.548 MB/s	0.546 MB/s	0.613 MB/s	+0.4%	-10.6%
bs=1000 sw=10 sl=64	p50	1,115,167 us	1,116,468 us	999,606 us	-0.1%	+11.6%
bs=1000 sw=10 sl=64	p95	1,140,207 us	1,155,929 us	1,046,770 us	-1.4%	+8.9%
bs=1000 sw=10 sl=64	p99	1,140,207 us	1,155,929 us	1,076,937 us	-1.4%	+5.9%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,566.80,200,128000,353,0.215,27596.18,38666.16,38666.16
1,100,10,64,20,2637.13,2000,1280000,758,0.463,124915.32,192393.68,192393.68
2,1000,10,64,20,22258.59,20000,12800000,899,0.548,1115167.28,1140207.37,1140207.37