Skip to content

feat(dataset): add contributor metadata storage and API - #6952

Open
xuang7 wants to merge 1 commit into
apache:mainfrom
xuang7:feat/dataset-contributor-backend
Open

feat(dataset): add contributor metadata storage and API#6952
xuang7 wants to merge 1 commit into
apache:mainfrom
xuang7:feat/dataset-contributor-backend

Conversation

@xuang7

@xuang7 xuang7 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

This PR adds contributor metadata to datasets so that dataset authors and other contributors can be properly acknowledged. Each dataset can record a list of contributors, where each contributor has a name, a creator flag, an email address, an affiliation, and a free-text comments field.

Changes:

  • Adds a new dataset_contributor table with a foreign key to dataset and cascade deletion in texera_ddl.sql, along with the incremental migration script sql/updates/29.sql and changelog entry 29.
  • Updates file-service so that contributors are:
    • returned with the dataset detail response,
    • accepted during dataset creation, and
    • replaceable through a new POST /dataset/{did}/update/contributors endpoint protected by the existing WRITE-access check.
  • Validates contributor names as required fields with a maximum length of 256 characters.

Currently, contributors are stored independently for each dataset. Cross-dataset contributor identity is left as a future extension. The UI will be added in a follow-up PR.

Any related issues, documentation, discussions?

Related to #6926

How was this PR tested?

7 ScalaTest cases were added.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Fable 5)

Add a dataset_contributor table (name, creator flag, email, affiliation,
comments) with cascade delete, expose contributors on the dataset detail
response, accept them at dataset creation, and add a write-guarded
POST /dataset/{did}/update/contributors endpoint that replaces the list.

Co-Authored-By: Claude Fable 5 <[email protected]>
@github-actions github-actions Bot added feature ddl-change Changes to the TexeraDB DDL platform Non-amber Scala service paths labels Jul 27, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @kunwp1, @Neilk1021
    You can notify them by mentioning @kunwp1, @Neilk1021 in a comment.

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.93%. Comparing base (8c7089b) to head (e3c36cb).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #6952      +/-   ##
============================================
- Coverage     78.93%   78.93%   -0.01%     
  Complexity     3784     3784              
============================================
  Files          1160     1160              
  Lines         46049    46047       -2     
  Branches       5110     5110              
============================================
- Hits          36350    36348       -2     
  Misses         8078     8078              
  Partials       1621     1621              
Flag Coverage Δ *Carryforward flag
access-control-service 70.00% <ø> (ø)
agent-service 76.76% <ø> (ø) Carriedforward from 8c7089b
amber 72.17% <ø> (ø) Carriedforward from 8c7089b
computing-unit-managing-service 20.49% <ø> (ø)
config-service 66.66% <ø> (ø)
file-service 67.07% <ø> (-0.14%) ⬇️ Carriedforward from 8c7089b
frontend 82.91% <ø> (ø) Carriedforward from 8c7089b
notebook-migration-service 78.94% <ø> (ø)
pyamber 95.38% <ø> (ø) Carriedforward from 8c7089b
workflow-compiling-service 26.31% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 5 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main 31f1e7b benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 383 0.234 25,567/33,728/33,728 us 🔴 +21.5% / 🔴 +107.8%
🟢 bs=100 sw=10 sl=64 833 0.508 118,235/133,709/133,709 us 🟢 -11.5% / 🔴 +23.1%
bs=1000 sw=10 sl=64 953 0.582 1,050,028/1,088,475/1,088,475 us ⚪ within ±5% / 🔴 -5.1%
Baseline details

Latest main 31f1e7b from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 383 tuples/sec 457 tuples/sec 767.9 tuples/sec -16.2% -50.1%
bs=10 sw=10 sl=64 MB/s 0.234 MB/s 0.279 MB/s 0.469 MB/s -16.1% -50.1%
bs=10 sw=10 sl=64 p50 25,567 us 21,330 us 12,502 us +19.9% +104.5%
bs=10 sw=10 sl=64 p95 33,728 us 27,764 us 16,234 us +21.5% +107.8%
bs=10 sw=10 sl=64 p99 33,728 us 27,764 us 18,919 us +21.5% +78.3%
bs=100 sw=10 sl=64 throughput 833 tuples/sec 836 tuples/sec 974.8 tuples/sec -0.4% -14.5%
bs=100 sw=10 sl=64 MB/s 0.508 MB/s 0.51 MB/s 0.595 MB/s -0.4% -14.6%
bs=100 sw=10 sl=64 p50 118,235 us 116,762 us 102,449 us +1.3% +15.4%
bs=100 sw=10 sl=64 p95 133,709 us 151,101 us 108,652 us -11.5% +23.1%
bs=100 sw=10 sl=64 p99 133,709 us 151,101 us 116,310 us -11.5% +15.0%
bs=1000 sw=10 sl=64 throughput 953 tuples/sec 956 tuples/sec 1,004 tuples/sec -0.3% -5.1%
bs=1000 sw=10 sl=64 MB/s 0.582 MB/s 0.583 MB/s 0.613 MB/s -0.2% -5.0%
bs=1000 sw=10 sl=64 p50 1,050,028 us 1,044,502 us 999,606 us +0.5% +5.0%
bs=1000 sw=10 sl=64 p95 1,088,475 us 1,095,818 us 1,046,770 us -0.7% +4.0%
bs=1000 sw=10 sl=64 p99 1,088,475 us 1,095,818 us 1,076,937 us -0.7% +1.1%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,522.35,200,128000,383,0.234,25566.56,33727.71,33727.71
1,100,10,64,20,2402.14,2000,1280000,833,0.508,118234.85,133709.45,133709.45
2,1000,10,64,20,20979.01,20000,12800000,953,0.582,1050028.12,1088474.63,1088474.63

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ddl-change Changes to the TexeraDB DDL feature platform Non-amber Scala service paths

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants