feat(dataset-mount): use mounted datasets in Python UDFs - #6897
Open
aicam wants to merge 3 commits into
Open
Conversation
Perform the FUSE mount for dataset repositories outside the (unprivileged) computing-unit pod. A per-node privileged `texera-mounter` DaemonSet runs GeeseFS on the pod's behalf and the read-only mount is exposed back into the pod via mount propagation, scoped to that computing unit. A JWT-authenticated S3 proxy in file-service fronts the LakeFS S3 gateway: it verifies the pod's JWT, checks the user's read access, and re-signs to LakeFS with credentials held only server-side, so no global credential ever enters the pod. - `texera-mounter` DaemonSet: mounter.py (+ tests), dockerfile, helm daemonset/rbac/values. - Unprivileged CU pod + mount-propagation wiring (KubernetesClient) and mounter config/env. - File-service JWT S3 proxy (S3ProxyServlet). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01H27sHgS29u2JcLYPPR6Hdx
Add per-computing-unit dataset mounting on top of the mount
infrastructure. A user can mount / list / unmount a dataset version on a
running computing unit; the platform asks that CU's node mounter to
perform the mount and reports the in-pod path back.
- ComputingUnitManagingResource: GET/POST/DELETE
/computing-unit/{cuid}/mounts, backed by a credential-free MounterClient
that talks to the node mounter over HTTP.
- FileResolver: resolve a dataset path to (repo, versionHash) and back,
so mounts can be shown to the user as /owner/dataset/version.
- "Mount datasets into computing unit" UI on the computing-unit selector.
At this point datasets can be mounted onto a CU but are not yet consumed
by any operator; that follows in the Python-UDF integration.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01H27sHgS29u2JcLYPPR6Hdx
Let a Python UDF bind mounted datasets to variables. Each binding maps a dataset version to a Python variable that, at runtime, holds the local filesystem path of the mounted dataset. - PythonUDFOpDescV2: a list of (variableName -> dataset) bindings, validated as Python identifiers and resolved via FileResolver; carried through PhysicalOp and WorkerConfig. - DatasetMountManager: on the executor, ensure each bound dataset is mounted on the computing unit and hand its path to the Python worker, which injects it as a module-level variable (MOUNTED_DATASETS). - "Mounted dataset variables" property editor for the Python UDF. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01H27sHgS29u2JcLYPPR6Hdx
github-actions
Bot
added
feature
engine
pyamber
frontend
Contributor
Automated Reviewer SuggestionsBased on the
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #6897 +/- ##
============================================
+ Coverage 77.30% 82.71% +5.40%
+ Complexity 3524 2353 -1171
============================================
Files 1161 894 -267
Lines 45922 37899 -8023
Branches 5101 3980 -1121
============================================
- Hits 35501 31348 -4153
+ Misses 8840 5412 -3428
+ Partials 1581 1139 -442
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 366 | 0.223 | 26,626/35,096/35,096 us | 🔴 -12.5% / 🔴 +116.2% |
| 🔴 | bs=100 sw=10 sl=64 | 759 | 0.463 | 131,565/150,560/150,560 us | 🔴 +7.7% / 🔴 +38.6% |
| ⚪ | bs=1000 sw=10 sl=64 | 881 | 0.537 | 1,129,918/1,222,594/1,222,594 us | ⚪ within ±5% / 🔴 +16.8% |
Baseline details
Latest main 49f9e2c from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 366 tuples/sec | 418 tuples/sec | 767.9 tuples/sec | -12.4% | -52.3% |
| bs=10 sw=10 sl=64 | MB/s | 0.223 MB/s | 0.255 MB/s | 0.469 MB/s | -12.5% | -52.4% |
| bs=10 sw=10 sl=64 | p50 | 26,626 us | 24,494 us | 12,502 us | +8.7% | +113.0% |
| bs=10 sw=10 sl=64 | p95 | 35,096 us | 35,449 us | 16,234 us | -1.0% | +116.2% |
| bs=10 sw=10 sl=64 | p99 | 35,096 us | 35,449 us | 18,919 us | -1.0% | +85.5% |
| bs=100 sw=10 sl=64 | throughput | 759 tuples/sec | 804 tuples/sec | 974.8 tuples/sec | -5.6% | -22.1% |
| bs=100 sw=10 sl=64 | MB/s | 0.463 MB/s | 0.491 MB/s | 0.595 MB/s | -5.7% | -22.2% |
| bs=100 sw=10 sl=64 | p50 | 131,565 us | 122,200 us | 102,449 us | +7.7% | +28.4% |
| bs=100 sw=10 sl=64 | p95 | 150,560 us | 152,961 us | 108,652 us | -1.6% | +38.6% |
| bs=100 sw=10 sl=64 | p99 | 150,560 us | 152,961 us | 116,310 us | -1.6% | +29.4% |
| bs=1000 sw=10 sl=64 | throughput | 881 tuples/sec | 894 tuples/sec | 1,004 tuples/sec | -1.5% | -12.2% |
| bs=1000 sw=10 sl=64 | MB/s | 0.537 MB/s | 0.545 MB/s | 0.613 MB/s | -1.5% | -12.4% |
| bs=1000 sw=10 sl=64 | p50 | 1,129,918 us | 1,118,836 us | 999,606 us | +1.0% | +13.0% |
| bs=1000 sw=10 sl=64 | p95 | 1,222,594 us | 1,182,930 us | 1,046,770 us | +3.4% | +16.8% |
| bs=1000 sw=10 sl=64 | p99 | 1,222,594 us | 1,182,930 us | 1,076,937 us | +3.4% | +13.5% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,547.13,200,128000,366,0.223,26626.46,35096.10,35096.10
1,100,10,64,20,2634.38,2000,1280000,759,0.463,131565.36,150560.39,150560.39
2,1000,10,64,20,22711.54,20000,12800000,881,0.537,1129918.32,1222594.34,1222594.34
Contributor
|
@aicam Please make the PR description more readable. Also this PR changed 44 files. Can we divide it? |
Contributor
Author
Its because its a PR part of a PR series:
When the preceding PR get merged, number of line changes update. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Let a Python UDF consume mounted datasets — the final step of the dataset-mounting feature (#6606), stacked on the per-computing-unit mounting PR (#6896).
Each UDF binding maps a dataset version to a Python variable that, at runtime, holds the local filesystem path of the mounted dataset, e.g.
open(f"{A}/file.csv").PythonUDFOpDescV2— a list of(variableName -> dataset)bindings, validated as Python identifiers and resolved viaFileResolver; the resolved locators flow throughPhysicalOpandWorkerConfig.DatasetMountManager— on the executor, ensures each bound dataset is mounted on the computing unit and hands its path to the Python worker, which injects it as a module-level variable (MOUNTED_DATASETS).Any related issues, documentation, discussions?
Closes #6895 · part of #6606 · builds on #6896.
How was this PR tested?
PythonUDFOpDescV2Spec(binding validation — Python identifiers, blank rows, empty dataset),DatasetVariableMappingSpec,DatasetMountManagerSpec(locator/identity validation),PythonWorkflowWorkerStartupConfigSpec, andtest_executor_manager/test_run_python_worker(theMOUNTED_DATASETSvariable injection).torch.load-ed it from the propagated path with bit-exact output.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8