feat(python): flows_to and the L3/L4 slice/flow verbs on the Python facade (#270, rc.1 leg) - #297
Open
rahlk wants to merge 25 commits into
Open
feat(python): flows_to and the L3/L4 slice/flow verbs on the Python facade (#270, rc.1 leg)#297rahlk wants to merge 25 commits into
rahlk wants to merge 25 commits into
Conversation
…40) (#274) * feat(cpg): _NullSafeBase + Span (#240) * feat(cpg): Edge + Import leaf models (#240) * test(cpg): cover Edge empty-prov default and Import.span default (#240) * feat(cpg): open-kind Node covering type/callable/body facets (#240) * feat(cpg): Module/Application/Analyzer/AnalysisPayload + exports (#240) * test(cpg): cover Application->Module->Node deep composition (#240) * test(cpg): parse real L1/L4 samples from both analyzers + superset gate (#240) * fix(cpg): enforce id on durable nodes positionally; body nodes exempt (#240) * test(cpg): pin the F7/F3 accessor contract against real L4 samples (#240) * test(cpg): genuinely pin module.functions accessor; strengthen TS source check (#240) * fix(cpg): add Module.span so the common span field parses (#240) span is listed as a common field on every node in the keystone (Part II), module included, but Module had no span field so it degraded to a raw dict in model_extra instead of parsing as Span. The ts-a4/ts-a1 fixtures emit span on the module node. * test(cpg): pin cdg/summary/k_limit/TS body+cfg in the accessor contract (#240) extra="allow" absorbs unknown keys, so deleting a canonical field leaves every parse test green — the accessor contract is the only guard. cdg and summary (both in F7's cfg/cdg/ddg/summary read set), the envelope k_limit, and TS callable body/cfg were unpinned. Each new assertion dereferences an element/field rather than doing a bare isinstance check, since a raw dict-of-dicts under extra="allow" still satisfies isinstance(list)/isinstance(dict) — confirmed by temporarily removing each field from the model and watching the new asserts fail before restoring.
* feat(graph): slice/flow result objects (#270) * fix(graph): serialize FlowResult paths in to_json (#270) * feat(graph): provider ABC seam + polymorphic resolve_vertex (#270) * feat(graph): capability gating with honest-degrade + strict (#270) * feat(graph): engine intraprocedural slices + control_deps (#270) * fix(graph): MultiDiGraph program graph + seed-consistent slices (#270) * feat(graph): flows_to witnesses + def_use with data-derived confidence (#270) * fix(graph): dedup flows_to witnesses over parallel edges; def_use seed-consistency (#270) * feat(graph): level-driven interprocedural slice depth (#270) * fix(graph): control_deps is intraprocedural — no sdg overlay at L4 (#270) * fix(graph): gate the sdg overlay on the ddg edge family (#270) _intra applied the sdg (dataflow) overlay whenever the backend was L4 and interprocedural was wanted, ignoring the edges family filter — so a cfg- or cdg-only slice picked up dataflow vertices from foreign callables. Fold the family gate into want_inter so the overlay predicate and explain()["interprocedural"] stay one source of truth: no dataflow family requested means no boundary crossing. Subsumes the redundant max_level()>=4 check on the overlay branch; control_deps' explicit interprocedural=False stays as belt-and-suspenders. * fix(graph): flows_to requires L4, honest-degrade to intra ddg at L3 (#270) flows_to's full semantics are ddg + summary/param_in/param_out — an interprocedural (L4) capability — but it gated on require(3), so an L3 backend silently returned intra-only results as if they were complete. Raise the requirement to L4: non-strict now attaches the degraded note (absence is UNKNOWN, not safety) while still returning the intra ddg witnesses it can compute; strict=True raises CapabilityError. * fix(graph): flows_to spans source and sink callables (#270) flows_to built its dataflow graph from the source's callable only, so a sink inside a different callable (reachable via param_in into the callee interior) was unreachable and reported a false "no flow". _dataflow_graph now unions the intra ddg graphs of the given callables before adding the sdg overlay, and flows_to passes both endpoint callables. def_use keeps its single-callable scope with a NOTE — interprocedural completeness lands with the whole-program dataflow graph (deferred to Task 7). * fix(graph): empty location resolution raises ValueError, not IndexError (#270) resolve_location legitimately returns [] when no vertex sits at the given line, but every engine verb indexes resolve_vertex(...)[0], turning an ordinary user miss into an IndexError. Raise a descriptive ValueError at the source in resolve_vertex instead, converting all [0] call sites into a clean failure mode. * fix(graph): MultiDiGraph annotations and provider contract docstrings (#270) The graph is an nx.MultiDiGraph everywhere, but three annotations still said nx.DiGraph: ProgramGraphProvider.program_graph, GraphResult.subgraph (and engine._dataflow_graph, already corrected in the C2 commit). Fix the first two and document the provider contract: parallel cfg/cdg/ddg edges between the same vertex pair must stay distinct edges with their own family/var/prov/kind, and sub-L3 providers still answer the structural methods — the engine handles level gating via require(...). * fix(graph): preserve sdg edge kind in flow witnesses (#270) The sdg overlay dropped e.kind (param_in/param_out/summary) on the floor in both _intra and _dataflow_graph, so a FlowPath hop crossing a callable boundary could only say "sdg" — not which boundary edge carried the flow. Carry kind on the overlay edges and have the hop dict prefer the edge kind over the family: intra hops still report cfg/cdg/ddg, boundary hops now report the concrete sdg kind. * fix(graph): per-verb evidence roles — control and use, not always def (#270) _evidence stamped every non-seed vertex "def", which is wrong for control_deps (the guard CONTROLS the seed) and def_use (downstream vertices are USES of the definition). Thread a default_role through _evidence and _intra: slices and flows keep "def", control_deps passes "control", def_use passes "use"; seeds are always "seed". * fix(graph): bound flows_to witness enumeration, report truncation (#270) flows_to enumerated simple paths with a hard-coded cutoff=64 and no path cap, and never told the caller when witnesses were dropped. Hoist the bounds to module constants (_PATH_CUTOFF=64, _MAX_PATHS=1000 — provisional, to be tuned on real graphs), stop collecting at _MAX_PATHS, and surface explain()["truncated"] so a partial witness set is never silently presented as complete. * fix(graph): _ddg_tier ranks by prov membership, not exact-list match (#270) prov is a provenance set in list form; the exact-list comparisons (prov == ["points-to"] / == ["ssa"]) made the STRONGER combined provenance ["ssa", "points-to"] fall through to "unresolved". Rank by membership: points-to anywhere means resolved, else ssa means structural, else unresolved.
… CanNode keys, fail-fast version check (closes #268)
Rewrite the read-only TS Neo4j backend to query the canonical 2.0.0 graph
(CanNode identity, TS-prefixed labels/relationships) and reject any other
schema on connect.
- Every Cypher query maps to the 2.0.0 vocabulary: :Symbol -> :CanNode,
:Callable/:Class/... -> :TSCallable/:TSClass/..., CALLS/DECLARES/HAS_METHOD/
RESOLVES_TO/HAS_MODULE -> TS_CALLS/TS_DECLARES/TS_HAS_METHOD/TS_RESOLVES_TO/
TS_HAS_MODULE. Application matched by `id ENDS WITH "/" + $app` (no `name`);
modules keyed by `_module` (no `file_key`).
- Call sites are now :TSBodyNode {kind:"call"} reached via TS_HAS_BODY_NODE and
resolved through TS_RESOLVES_TO (no :CallSite nodes); reconstruct reads the
target from `callee`.
- Fail-fast: `_check_schema_version` reads (:Application).schema_version once in
__init__ and raises the new CldkSchemaMismatchException unless it is "2.0.0".
- Accessors whose vocabulary 2.0.0 does not project raise NotImplementedError
(no JSON fallback exists for a read-only Cypher client): get_decorators,
get_class_decorators, get_methods_with_decorators, get_classes_with_decorators,
get_all_fields, get_interface_properties, get_imports, get_all_exports,
get_all_variables. Whole-object reconstruction degrades those sub-fields to
empty so graph-supported accessors keep working. Public facade names and
return types are unchanged.
- Tests: new test_typescript_neo4j_schema.py covers the version gate and the
fallback raises without a live graph; the stubbed-graph parity tests adopt the
2.0.0 query strings; the live-graph integration suite is skipped until the
analyzer pin moves to >=1.0.0 (Task 9).
…rify assumptions (#268)
Review fixes on the graph-schema-2.0.0 rewrite:
- `_resolve_application_id` resolves `application_name` to exactly one
:Application `can://` id on connect; 0 or >1 suffix matches raise
CodeanalyzerUsageException naming the candidate ids, so a shared database
with two apps ending in the same path segment can no longer silently merge
module scopes. `_load_module_keys` anchors on the resolved id. The schema
check still runs first (DB-level) so pre-2.0.0 graphs keep the actionable
mismatch error; rationale commented in __init__.
- Every unverified 2.0.0 assumption now carries a VERIFY(2.0.0-e2e) marker at
its query site (`_module` scoping, array-prop class hierarchy, method_name
fallback, reconstruct's module_name->name fallback); the module docstring
states the `_module` claim as to-verify, and the two leftover v1
"Symbol-keyed" prose comments are reworded.
- Query-construction unit tests capture the built Cypher through the `_run`
seam for the two riskiest rewrites: the TS_HAS_BODY_NODE / TSBodyNode
{kind:"call"} / TS_RESOLVES_TO call-site path (reading `callee`) and
TSModule `_module` scoping, plus the app-id guard (0-match and 2-match).
- CldkSchemaMismatchException exported from cldk.utils.exceptions __all__.
…all-graph keys, fail-fast version checks Adapt the Python facade to codeanalyzer-python 1.0.x (analysis schema 2.0.0): - unwrap the v2 Analysis envelope in _run_analyzer and fail fast unless schema_version is the SUPPORTED_ANALYSIS_SCHEMA (2.0.0) - follow the model renames everywhere the backends walk the symbol table: module.classes→types, class methods→callables, inner_classes→types, callable inner_callables→callables, inner_classes→types - recover callable source by slicing module.source with span.bytes (_code_of): schema v2 stores source once per module instead of a code field per callable - translate CanNode can:// call-edge endpoints (PyCallEdge.src/dst) back to dotted signatures for the public call graph; externals keep their raw ids - read the renamed PY_CALLS edge property (prov) and construct PyCallEdge with src/dst/prov in the Neo4j backend - add a fail-fast schema_version gate to PyNeo4jBackend, reading the stamp on the scoped :PyApplication node (mirrors TSNeo4jBackend) - map reconstruct.py's graph-vocabulary kwargs (methods/inner_*) onto the new model fields; the callable's code node property is no longer reconstructed - pin codeanalyzer-python==1.0.2 (1.0.0/1.0.1 emitter fix in #104 verified) and codeanalyzer-typescript==1.0.0 in dependencies and [tool.backend-versions] - rewrite the hand-built test fixtures in the new vocabulary and add test_python_schema_contract.py: both fail-fast gates + CanNode translation
…S.md) Record the locked design decisions for the Rust query engine (epic #279): M1 scope, can:// identity, fresh plan-algebra ADR, serde+Bolt data plane, Opaque plan-split, fat-wheel packaging, cldk.graph slicer replacement, and the root crates/ workspace layout. Repo-level gitignore exception overrides the global .claude ignore for this one file.
, #290, #292) (#294) Ports the fixes shipped on main to the 2.0 line (2.0.0-rc.1 shipped jarless): - Force-include the codeanalyzer-java JAR via [tool.hatch.build] artifacts (#284) - Pipe-safe release guard that fails a jarless build before publishing (#284) - Source release notes from CHANGELOG.md and fail on blank; drop the label-based changelog scraper and orphaned release_config.json (#289) - Remove the obsolete native-binary test and its unused import (#290) - Fix the stale conftest docstring for the retired native binary (#292) Verified: build on this branch (with .git) yields a 32MB wheel+sdist containing the JAR; tests/analysis/java/test_jcodeanalyzer.py -> 42 passed.
…ardening + pyfix goldens (#270)
…stead of crashing (#270)
…ds (closes #295) (#270)
…al provider (#270) _index() walked only mod.functions and one level of mod.types[*].callables, so closures (a callable's own nested callables) and nested classes (classes inside classes or inside callables) were invisible to program_graph/callable_of/resolve_location — silently degrading to seed-only with no note, even though the Neo4j emitter and the analyzer's own call-graph walker both recurse fully. _add/_add_type now recurse into getattr(..., "callables"/"types", None) defensively, so every callable that owns a body ends up indexed regardless of nesting depth, while staying a no-op on model families (e.g. the cldk-native cpg.models.Node) that don't support nested classes at all. Also fixes two vacuous assertions in the same test file surfaced alongside this change: a `... or True` tautology, and an `and`/`or` precedence bug that made a source-slice assertion pass regardless of its left operand.
…_slice contract (#270)
resolve_location: the local mixin accepts a basename/suffix seed (path.endswith("/" +
file)) in addition to an exact module-path match, and a golden test already pins this; the
Neo4j backend only matched exactly, so the same seed string behaved differently per
backend. Cypher now reads (n._module = $file OR n._module ENDS WITH $suffix).
source_slice: aligned to the adjudicated three-way contract (vertex exists without a
span/start_line -> (module_path, None); vertex unknown/not found -> (None, None); never
fabricate a line number by parsing the vertex key's shape). The previous implementation
degraded synthetic ports (@entry, @formal_in:N, ...) to (None, None) instead of
(module_path, None), and worse, could fabricate a plausible-looking "module:line" for a
vertex that doesn't exist at all whenever its key happened to parse as line:col shape.
Rewritten to resolve the exact vertex by comparing _to_uri(n.id) against the requested
vertex_uri, scoped to the owning callable's PyCFGNode rows (one round trip).
sdg_edges: scoped the destination side too (b._module IN mods), matching the existing
source-side scoping and every other query in this backend.
…n, flows_to, slice_forward (#270) program_graph parity now includes prov (as a tuple) in the edge-comparison key, so a backend that dropped or reordered ddg provenance no longer passes silently. Two new tests assert full-coverage parity rather than one hand-picked seed per verb: source_slice/callable_of agreement for every vertex of every callable, and resolve_location's full hit-list agreement for every (file, line) that owns a real vertex. test_verb_parity now also drives flows_to (fed a real param_in pair discovered dynamically from the live application, since the fixture's app_name is a random tmp-dir name each run) and slice_forward, alongside the existing slice_backward/ def_use/control_deps. Also corrects the module docstring's claim that the analyzer's bolt writer prunes other applications globally per emit -- it scopes the orphan-module prune to this application's own app_name; the dedicated-container advice is kept for run isolation, not because of any cross-application risk.
…comments (#270) max_level capture is now unconditional (self._max_level = analysis.max_level, no hasattr guard) and max_level() drops its getattr(..., 1) fallback -- an envelope that somehow lacks max_level now fails loudly instead of silently reporting level 1. _StubAnalysis in the schema-contract test gains a max_level default to match. Kept the hasattr(self, "_level_int") guard around opts["analysis_level"]: verified it IS load-bearing -- test_python_schema_contract.py's _bare_local_backend() exercises _run_analyzer on a __new__ instance that never set _level_int, to isolate the schema-envelope gate from analyzer construction. Added a comment naming that test as the reason instead of removing it. Dropped the unused json/Path imports from test_facade_delegates.py.
Drop-in compatible per the 2026-07-27 e2e validation: schema 2.0.0 unchanged, prov vocabulary unchanged (scalpel default L4 oracle emits the same ssa/points-to), fixture-identical output on the pyfix sample, live dual-backend parity 6/6, real-repo smoke 8/8 old + 6/6 new API.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #270 (rc.1 leg of the staged rollout — rc.2 TS, rc.3 Java, rc.4 Go, rc.5 C++, 2.0.0 Rust engine follow).
What this adds
Five verbs on
PythonAnalysis, backed by the shared engine merged in #271, on both backends:slice_backward/slice_forward/control_deps/def_use/flows_to(interprocedural at L4, witness paths with per-hop kind/var/confidence)AnalysisLevelmaps to the analyzer's-a 1..4;max_levelis recorded from the same in-process run (never sniffed); underscore level names ("call_graph") accepted alongside enum valuesCpgLocalProviderMixin: language-neutral local provider over the (schema-2.0-shaped) upstream models — mintscan://<callable>@<key>vertex URIs matching the analyzers' ownparam_in/param_outvocabulary; recurses into closures and inner-class methodsPyNeo4jBackend: the five provider primitives in Cypher over thePY_*L3/L4 overlay, speaking the emitter's real can:// vertex ids (closes #295 found mid-branch by the live parity suite)cldk.graphpublic exports; capability gate degrades honestly below level floors,strict=TrueraisesVerification
tests/graph+tests/analysis/python(exact-set goldens hand-derived from the committedpy-a4.jsonL4 sample)cldk/itself at-a 4: 8/8 legacy accessors, 6/6 new-verb calls, genuine cross-callableflows_towitness, honest degrade at L2Known deferrals (rc.2 touch-up candidates)
resolve_locationhas the Cypher fix but no discriminating stub test (stub ignores params; live suite only probes full paths)_MODULES_CTErecomputes app scoping per call instead of reusing cachedself._modules_index()(no public callable-enumerator yet)max_levelon thePyApplicationnode; SDK currently derives it from overlay presence)#270 stays open for the remaining legs.