Add bundle bitmap for per-field telemetry - #6104
Draft
denik wants to merge 8 commits into
Draft
Conversation
Co-authored-by: Isaac
Encode presence of every bundle config field as one bit. The schema is derived by reflecting over config.Root, recorded in bundle/bitmap/schema.txt, and embedded into the binary. Append-only so bit positions stay stable. Adds hidden `databricks bundle bitmap` group: schema, update-schema [--validate], bitmap-text, bitmap. Wires generate-bitmap-schema into the Taskfile and runs the encoder over all no_drift invariant configs. Co-authored-by: Isaac
Kept in PR history for context; not landed. Co-authored-by: Isaac
- Every bool/*bool field now gets a "#true" value bit in addition to its presence bit, so a *bool encodes unset/false/true as 00/10/11. - decode_bitmap.py decodes the base64 bitmap against the schema; the dedicated acceptance test asserts it round-trips to bitmap-text. - Record the encoded bitmap and set-bit list for the default-python template. Co-authored-by: Isaac
Replace the string-keyed BoolValues map with a typed bundle.Telemetry struct whose fields feed both the deploy telemetry wire (via BoolValues()) and the bundle bitmap (namespaced under telemetry.*). Value-bearing flags whose false is meaningful are stored as True/False pairs so the bitmap can represent both outcomes; BoolValues() collapses each pair back to its single wire key. libs/cache gets a concrete Metrics struct (embedded into Telemetry) instead of the old interface. The wire is unchanged except that redundant false entries of one-hot groups (state_path_*, dms_undeclared_*) are no longer emitted; their true member is preserved. Co-authored-by: Isaac
Co-authored-by: Isaac
denik
force-pushed
the
denik/bundle-bitmap
branch
from
July 30, 2026 10:11
221d417 to
39faace
Compare
decode_bitmap.py was committed without the executable bit, so the round-trip step failed with "permission denied" (exit code differed across OSes, breaking the golden). Mark it executable. bitmap-text and bitmap are separate invocations sharing a cache dir, so the second recorded a cache hit instead of a miss and the round-trip check diverged. Give each its own cache dir so both observe a cold miss. Co-authored-by: Isaac
On a CRLF checkout (Windows) the embedded schema.txt lines kept a trailing \r, so they never matched the clean paths from WalkType. Merge's dedup then failed and the schema doubled, breaking bundle bitmap schema/bitmap-text on Windows. splitLines now normalizes CRLF. Co-authored-by: Isaac
Collaborator
Integration test reportCommit: a81474e
10 interesting tests: 5 RECOVERED, 3 SKIP, 2 flaky
Top 49 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Today each tracked DABs feature needs custom code (a key constant plus a SetBoolValue call from a mutator), so only hand-picked features are tracked. The bundle bitmap encodes the presence of every bundle config field as one bit, derived by reflecting over config.Root, so all current and future fields are covered for free (http://go/dabs-telemetry).
Changes
databricks bundle bitmapgroup:schema,bitmap-text,bitmap(compressed + base64), andupdate-schema [--validate]. The schema is recorded in bundle/bitmap/schema.txt, embedded into the binary, and append-only so bit positions stay stable across releases.task generate/generate-check, and the encoded bitmap is exercised over every no_drift invariant config.The deploy telemetry wire is unchanged except that redundant false entries of one-hot groups (state_path_, dms_undeclared_) are no longer emitted; their true member is preserved.