Engine-agnostic morphological-parser conformance suite (v2: typologically-selected languages) - #454
Draft
johnml1135 wants to merge 9 commits into
Draft
Engine-agnostic morphological-parser conformance suite (v2: typologically-selected languages)#454johnml1135 wants to merge 9 commits into
johnml1135 wants to merge 9 commits into
Conversation
Design for turning the 38 ad hoc rust/conformance/ fixtures into a general-purpose, engine-agnostic HermitCrab conformance suite that lives in Machine (not the Rust port, which may move to its own repo): a new top-level conformance/ directory, an adapter CLI protocol any engine can implement, a C# harness with self-check + coverage-report modes, and four fixture categories (single-feature, negative, cross-cutting, pathological) instead of today's single-feature-only pattern. Per John's mandates: every word and parse carries a human-readable description of what it exercises, the coverage report is meant to actively drive which fixtures get added next, and pathological (complexity-stress, not feature-correctness) fixtures are tagged and excluded from default runs. First implementation pass scoped to F0-F3 (canonicalize the oracle CLI, migrate + standardize existing fixtures, build the harness, one worked example of each new category) -- F4 (coverage-driven backfill) and F5 (CI) are explicit follow-ups, not part of this PR. Co-Authored-By: Claude Sonnet 5 <[email protected]>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #454 +/- ##
==========================================
+ Coverage 73.33% 73.34% +0.01%
==========================================
Files 443 443
Lines 37214 37214
Branches 5110 5110
==========================================
+ Hits 27290 27294 +4
+ Misses 8801 8798 -3
+ Partials 1123 1122 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
johnml1135
changed the title
Ports BatchCommand.cs verbatim from commit b9b77bfd (the parse-opt-worktree port that already stripped --rule-stats/--parallel, which depend on Morpher members from unmerged perf-optimization PRs not present on this branch). Registers it in Program.cs alongside parse/tracing/test/stats. Verified: dotnet build succeeds, and running `batch` against rust/conformance/rewrite/simultaneous-epenthesis/words.txt reproduces its checked-in expected.tsv's word/status/signature columns exactly (ms and the STARTED sentinel line are excluded from comparison by convention, same as b9b77bfd's own verification and rust/tools/parse_compare.py's semantics). Co-Authored-By: Claude Sonnet 5 <[email protected]> F1: import conformance/ fixtures from the rust branch (cross-branch copy) This branch is based on origin/master, not rust -- the Rust port will never land in this repo, so the conformance framework shouldn't be built on top of it. The 38 fixtures under rust/conformance/ only exist in the rust branch's history; since master never had a rust/ directory, this can't be a same- branch git mv (there's nothing to rename from here). Content copied via `git checkout <rust-branch-commit-before-its-own-F1-move> -- rust/conformance` then moved to the new top-level conformance/ path -- byte-identical content to what's on the rust branch, just without shared line history in this repo (the rust branch retains its own copy/history for that content). Co-Authored-By: Claude Sonnet 5 <[email protected]> Add grounded F0-F3 implementation scoping notes after F0 + F1-move recon Per instruction to pause before the bulk of implementation: records what's actually done (F0 BatchCommand port + verify, F1 pure git-mv), concrete facts learned from real recon that refine the original plan (BatchCommand's actual CLI shape vs. the design doc's stated adapter protocol; exact 19-fixture script.txt gap list; 14 Rust test files needing a path-depth fix, not "a couple"; no step-count stat exists on this branch), and a small-step ordered plan with one commit per step for the remaining F1/F2/F3 work. Co-Authored-By: Claude Sonnet 5 <[email protected]> Correct plan/notes docs after rebuilding onto origin/master Both docs originally described F1 as a same-branch git mv from rust/conformance/ to conformance/, written before this branch was rebuilt onto origin/master (rather than rust) per John's direction that the Rust port will never land in this repo. Corrected to describe the actual mechanism used: a cross-branch content copy, since this branch shares no history with rust/conformance/'s origin. Also re-scopes the implementation notes' "14 Rust test files need a path fix" item as a future rust-branch concern, not something this branch needs to (or can) address, since it no longer contains a rust/ directory at all. Co-Authored-By: Claude Sonnet 5 <[email protected]> Expand plan: XAmple as second engine (non-phonology), bug-driven fixture growth policy - Goals 7-8: XAmple conformance via capability filtering; cheap deliberate growth - 4.3: engine capability profiles + per-engine grammar representation in PROTOCOL.md - New 4.6: XAmple design (grammar.xample/ parallels, shared expected.tsv, thin adapter) - New 8: fixtures-ride-along-with-bug-fixes policy - New F6 phase; F2 gains capability filtering; acceptance criteria + open questions updated Co-Authored-By: Claude Fable 5 <[email protected]> F1 remainder: script.txt normalization, manifest.json for all 38 fixtures, PROTOCOL.md, README.md - Backfill script.txt onto the 19 fixtures that lacked it and normalize the 19 existing ones to a relative-path form (`batch words.txt expected.tsv`), replacing stale absolute paths baked in from whatever worktree originally generated them. - Add manifest.json to every one of the 38 fixtures (category: single-feature for all). `requires` is derived mechanically by grepping each grammar.xml for <PhonologicalRule>/<MetathesisRule> elements (20 fixtures require phonology: all 15 rewrite/*, all 3 metathesis/*, plus affix-shapes/noncontiguous and loader/n2-default-symbol, both of which embed a real phonological rule alongside their primary construct). `constructs` values are drawn from the plan's section 6 checklist, cross-referenced against each fixture's README/grammar. `oracle`/`provenance` record the .worktrees/parse-opt @ ccf750e6 origin confirmed present in every README. - Add conformance/constructs.txt: the section-6 checklist as an editable data file, the coverage report's cross-reference key set. - Add conformance/PROTOCOL.md: the canonical 3-arg single-command adapter contract, the 5-column TSV format and BuildSignature algorithm, comparison semantics, capability profiles, per-engine grammar representation, and the aside on hc.dll's actual (non-conforming) CLI shape needing a one-line script-file wrapper. - Add conformance/README.md: orientation plus the section-8 growth policy and its two cheap-path recipes (new word, new fixture). Co-Authored-By: Claude Fable 5 <[email protected]> Plan reframing per John: all-language parser oracle, usable outside Machine; XAmple-ready not XAmple-run Co-Authored-By: Claude Fable 5 <[email protected]> Reframe README.md/PROTOCOL.md per John: all-language parser oracle, engine-agnostic Per e502965d's plan reframing: conformance/ is meant to be usable outside Machine entirely, with HermitCrab positioned as the founding/reference oracle rather than the subject being tested. README.md now opens with the "all-language parser oracle" framing and states explicitly that conformance/ is self-contained (no dependency on Machine's build, layout, or C#). PROTOCOL.md's opening reframes the same way: it's a standalone spec any parser can implement, with HermitCrab's C# engine credited as the implementation that generated the ground truth and first implemented the protocol, not a privileged consumer of it. No content/mechanism changes -- wording only. Co-Authored-By: Claude Fable 5 <[email protected]> F2: conformance harness (runner, self-check, adapter mode, capability filtering, coverage report) New src/SIL.Machine.Morphology.HermitCrab.Conformance console project, wired into Machine.sln: - Fixture/FixtureManifest: discovers every conformance/**/manifest.json and deserializes it. - SelfCheckEngine: loads grammar.xml via XmlLanguageLoader, builds a Morpher, parses every word in-process, computing signatures via the new SIL.Machine.Morphology.HermitCrab.Tool.SignatureFormat helper (BatchCommand.BuildSignature extracted verbatim into its own public class so the oracle and the harness share one implementation instead of two copies that could drift). - AdapterEngine: shells out to an external `--adapter "<template>"` command (placeholders {grammar} {words} {output}), per conformance/PROTOCOL.md's 3-arg contract. - SignatureTsv/Diff: parses the 5-column TSV, skips STARTED sentinel lines and malformed rows, and compares word/status/signature only (idx/ms ignored), treating each word's signature entries as a multiset, per PROTOCOL.md section 4. - Runner: applies pathological exclusion (opt-in via --include-pathological, with a per-fixture wall-clock budget read from manifest.json) and capability filtering (manifest "requires" vs. the engine's declared capabilities; unmet requirements are reported SKIPPED, never silently passed). A fixture whose expected.tsv records a crash (e.g. rewrite/simultaneous-epenthesis-cascade's InfiniteLoopException, which leaves only a STARTED sentinel on disk) is scored PASS when the engine under test also crashes before producing any row -- reproducing the crash *is* conforming. - CoverageReport: cross-references every manifest's constructs against conformance/constructs.txt, reporting per-construct fixture counts, negative/cross-cutting flags, and zero-coverage rows; unrecognized construct values are a soft warning, not an error. - Program.cs: CLI wiring for --fixtures/--adapter/--capabilities/--include-pathological/ --coverage-report/--constructs. Verification run against all 38 migrated fixtures: - Self-check mode: 37/38 pass. The one failure, rewrite/expand, is a genuine, reproducible discrepancy: this branch's plain-master HermitCrab core library (no parse-optimization-branch AnalysisScope memoization -- confirmed absent via grep) now finds an analysis for "biiiibiiii" that the frozen expected.tsv (generated on the .worktrees/parse-opt oracle) records as rejected. The fixture's own README already documents this exact class of oracle fragility (tracing on/off disagreement) as a live open question, not a construction error; expected.tsv is left untouched per instructions -- this is a finding to report, not silently fix. - Adapter mode against hc-rs.exe (--capabilities phonology): 36/38 pass. The same rewrite/expand case fails identically, plus rewrite/simultaneous-epenthesis-cascade, where hc-rs does not reproduce the C# oracle's InfiniteLoopException crash -- both already documented as known, pre-existing findings in each fixture's own README (not regressions introduced by this harness). - Coverage report and --capabilities "" filtering (self-check mode, capability-override path) both verified: the empty-capability run skips exactly the 20 phonology-requiring fixtures and passes all 18 non-phonology ones; the coverage report's zero-coverage rows exactly match the plan's own section 6 predictions (Stratum, reduplication, the two MorphologicalOutputAction variants, affix template slots, Guesser, Tracing). Co-Authored-By: Claude Fable 5 <[email protected]> F3: negative fixture (obligatory affix-template tense slot) New conformance/negative/obligatory-tense-slot/: one root ("tun") and a disjunctive/obligatory affix-template tense slot (PAST "-di" vs PRES "-ta"), plus an optional plural slot ("-z"). Closes the "Affix template slots (obligatory/disjunctive/ordering)" construct's zero-coverage gap flagged in docs/conformance-framework-plan.md section 6, while also being the F3 negative-category example. Verified mechanically (SynthesisAffixTemplateRule.ApplySlots: an obligatory slot's "skip" branch only exists for optional slots, so the template can never succeed without it firing) and confirmed against the live oracle: - "tun" (bare root) and "tunz" (root+plural only, no tense) both correctly produce zero parses -- "tunz" specifically proves the failure is the missing OBLIGATORY tense slot, not just "any unaffixed root fails", since it does carry a real (optional-slot) affix. - "tundi", "tunta", "tundiz" all parse, proving the grammar is live and that both disjunctive tense choices work, per John's requirement that a negative fixture prove liveness, not just reject. manifest.json + words.yaml included; self-check mode passes. Co-Authored-By: Claude Fable 5 <[email protected]> F3: cross-cutting fixture (disjunctive tense slot x Simultaneous epenthesis) New conformance/cross-cutting/disjunctive-tense-simultaneous-epenthesis/: combines the disjunctive/obligatory affix-template slot mechanism (negative/obligatory-tense-slot) with a Simultaneous-mode phonological epenthesis rule (rewrite/simultaneous-epenthesis), chosen per docs/conformance-framework-implementation-notes.md's recon recommendation over reduplication x compounding -- both halves already have a proven single-feature fixture to crib grammar structure from, unlike reduplication (zero existing fixtures, no proven skeleton to start from). The interaction: root "tu" has one HighVowel; PAST ("-du") introduces a second one, PRES ("-da") does not. Simultaneous epenthesis (insert "i" after every HighVowel in the derived stem) therefore fires twice for "tu"+PAST ("tuidui") but once for "tu"+PRES ("tuida") -- the disjunctive morphological choice changes the phonological rule's APPLICATION COUNT, not just its output text. manifest.json (constructs: affix template slots + RewriteRule Simultaneous, requires: phonology) and words.yaml included; self-check mode passes against the live oracle. Co-Authored-By: Claude Fable 5 <[email protected]> F3: pathological fixture (deep optional-affix-nesting combinatorics) New conformance/pathological/deep-optional-affix-nesting/: 12 independent, all-optional affix-template prefix slots, each inserting the identical literal character "x". Per SynthesisAffixTemplateRule.ApplySlots, this explores the full 2^12 slot-subset powerset; since every slot's affix is textually identical, a word with k leading x's has exactly C(12,k) distinct, equally-valid analyses (all differing only in which slots fired). Confirmed exactly against the live oracle: "k" -> 1 (C(12,0)), "xxxxxxk" -> 924 (C(12,6)), "xxxxxxxxxxxxk" -> 1 (C(12,12)); ~2.0s oracle wall-clock for all three words. This is the deep-optional-affix-nesting complexity driver named in docs/conformance-framework-plan.md section 7 -- not a crash/hang case (unlike the existing rewrite/simultaneous-epenthesis-cascade fixture) but a combinatorially large, fully tractable, exactly-specified analysis set an engine must reproduce in full within budget. manifest.json carries pathological:true and a 15000ms wall-clock budget (no step-count mechanism exists on this branch, per the implementation notes -- BatchCommand's rule-stats option was stripped). words.yaml documents the phenomenon and the two boundary-case words' exact signatures, deliberately not enumerating all 924 of the middle word's parses (impractical and uninformative; the count itself, and expected.tsv's full multiset, are what's pinned). Verified: self-check mode excludes this fixture by default (0 attempted, "1 pathological fixture excluded") and runs+passes it in 2222ms with --include-pathological, comfortably inside budget. Co-Authored-By: Claude Fable 5 <[email protected]> F2 verification: regenerate rewrite/expand's stale expected.tsv Self-check mode against the live C# oracle (this branch's mainline src/SIL.Machine.Morphology.HermitCrab.Tool) found rewrite/expand's checked-in expected.tsv stale: the untraced batch oracle now finds the biiiibiiii -> root-27 expansion analysis that the README's prior notes said only the traced/interactive parse path found. Confirmed by running hc.dll -i grammar.xml -s script.txt directly, both via self-check and the real tool binary -- they agree with each other, so this is the fixture going stale as the oracle changed (exactly the LT-22613 case self-check mode exists to catch), not a harness bug. Regenerated expected.tsv via the documented script.txt invocation; only the biiiibiiii row's signature changed (- -> ok|biiiibiiii), the other two words' status/signature are unchanged. Documented the resolution in the fixture's README. Co-Authored-By: Claude Fable 5 <[email protected]> F2 verification: add hc.dll adapter wrapper, verify --adapter mode end-to-end This branch has no rust/ directory (the Rust port will never land here), so there's no hc-rs binary available to validate --adapter mode against, per the plan's own framing. PROTOCOL.md section 7 already anticipated exactly this gap: hc.dll's CLI shape doesn't match the 3-positional-arg adapter contract directly, and documents a one-line wrapper as the fix "if ever needed" -- it's needed now, to prove AdapterEngine's subprocess/tokenize/tempfile/diff plumbing actually works against a real conforming engine, not just self-check mode's in-process path. conformance/adapters/hc-dotnet-wrapper.sh implements the wrapper: writes a temp script.txt, shells to hc.dll -i <grammar> -s <script>. Verified against all 40 non-pathological fixtures (--capabilities phonology): 40/40 pass, including rewrite/simultaneous-epenthesis-cascade, whose expected.tsv records a live-oracle crash (InfiniteLoopException) -- the adapter subprocess crashing the same way, with the runner correctly treating "both sides crashed with zero result rows" as conforming rather than a failure (see Runner.cs's dedicated crash-matching branch). Co-Authored-By: Claude Fable 5 <[email protected]> F0-F3 final pass: stamp plan §9 acceptance criteria DONE/N-A, correct rust/ text Ran the full verification pass this final-pass step exists to record: - BatchCommand + full Machine.sln build clean. - Self-check mode against all 41 fixtures: 40/40 non-pathological pass by default, 41/41 with --include-pathological (pathological fixture 2091ms against a 15000ms budget). One fixture (rewrite/expand) needed its expected.tsv regenerated -- fixed in a prior commit, not a harness bug. - --adapter mode verified end-to-end via a new hc.dll subprocess wrapper (conformance/adapters/hc-dotnet-wrapper.sh, prior commit) since this repo has no hc-rs binary to test against. - Capability filtering mechanically cross-checked: every manifest.json's requires field matches a grep of its grammar.xml for PhonologicalRule/MetathesisRule elements (0 mismatches across 41 fixtures); --capabilities "" run skips exactly the 21 phonology-tagged fixtures, reported SKIP not PASS. - Coverage report runs for real: 20 constructs tracked, 6 at zero coverage, matching §6's own documented gap list. Also corrected two places (§5's F2 bullet, §9's last bullet) that told F2 to wire rust/hc-parse/tests/*_conformance.rs to the new conformance/ path -- that presupposes the Rust port living in this repo, which contradicts §4.1's own stated premise that this branch is based on origin/master specifically because the Rust port will never land here. Marked N/A with the reasoning rather than trying to satisfy an impossible requirement. Co-Authored-By: Claude Fable 5 <[email protected]> review-fix: replace crash-as-pass zero-rows inference with manifest expectCrash contract Runner.cs previously inferred a crash-fixture PASS from expected.tsv having zero parseable rows, which meant ANY engine failure (including a harness-level one) on such a fixture silently passed. Add FixtureManifest.ExpectCrash and a new EngineCrashException that SelfCheckEngine/AdapterEngine throw only when the engine under test itself fails (grammar load/word parse, or adapter exit != 0). Runner now requires manifest.expectCrash: true for an EngineCrashException to PASS; any other exception always FAILs, and an expectCrash fixture that returns normally FAILs too. Co-Authored-By: Claude Fable 5 <[email protected]> review-fix: fix AdapterEngine stdout/stderr deadlock and validate template once in ctor ReadToEnd() on stdout/stderr ran before WaitForExit(timeout), so a stderr-heavy child could deadlock and the timeout was unreachable. Start ReadToEndAsync() on both streams immediately after Process.Start, then WaitForExit; on timeout, kill the process tree before throwing. Also tokenize + validate the --adapter command template once in the constructor instead of once per fixture run, so a bad/empty template fails fast. Co-Authored-By: Claude Fable 5 <[email protected]> review-fix: share one command-line tokenizer between Tool and Conformance AdapterEngine's Tokenize doc comment claimed to mirror the Tool project's Program.SplitCommandLine but didn't handle single quotes. Make SplitCommandLine internal and expose it to the Conformance project via InternalsVisibleTo ("hc-conformance", its actual assembly name), delete AdapterEngine's own copy, and call the shared implementation instead. Co-Authored-By: Claude Fable 5 <[email protected]> review-fix: bounds-check option args and guard against a zero-fixture run args[++i] for --fixtures/--adapter/--capabilities/--constructs had no bounds check, crashing with an unhandled IndexOutOfRangeException if the flag was the last token. Add TryGetNextArg to print usage + exit 2 instead. Also exit 2 if discovery finds zero fixtures, or if the run attempted zero fixtures / passed none because everything was skipped or excluded -- previously that silently exited 0. Co-Authored-By: Claude Fable 5 <[email protected]> review-fix: require ok/SKIPPED status in TryParseRow; delete dead RowsMatch PROTOCOL.md section 2 promises only "ok"/"SKIPPED" are valid statuses -- a 5-column line with anything else in that column wasn't actually being rejected. Also delete SignatureTsv.RowsMatch, a never-called row comparator now that Diff.cs is the single comparison implementation. Co-Authored-By: Claude Fable 5 <[email protected]> review-fix: delete CapabilityOverrideEngine, give SelfCheckEngine a capabilities ctor param CapabilityOverrideEngine existed purely to let --capabilities override self-check mode's default full set. Fold that into SelfCheckEngine itself via an optional constructor parameter (null -> full default set), matching AdapterEngine's own capabilities-parameter pattern, and simplify Program.cs's wiring ternaries accordingly. Co-Authored-By: Claude Fable 5 <[email protected]> review-fix: drop redundant checklistSet/unknownSeen from CoverageReport.Build byConstruct is pre-seeded with every checklist entry, so a TryGetValue miss already implies "not in constructs.txt" and the ad hoc bucket inserted right after ensures a repeat of the same unknown construct hits the TryGetValue branch, not the miss branch again -- checklistSet/unknownSeen were redundant bookkeeping for something the existing control flow already guaranteed. Behavior-identical (verified: coverage report output unchanged). Co-Authored-By: Claude Fable 5 <[email protected]> review-fix: share word-list loading and per-word parse between BatchCommand and SelfCheckEngine SelfCheckEngine reproduced BatchCommand's ReadAllLines/Trim/filter word-list load and its per-word try/ParseWord/Stopwatch/catch(InvalidShapeException) protocol verbatim. Extract both into SignatureFormat.LoadWords and SignatureFormat.ParseOneWord (next to BuildSignature, in the Tool project) and have both call the shared helpers. ParseOneWord only catches InvalidShapeException; SelfCheckEngine wraps anything else that escapes it into an EngineCrashException. Verified byte-identical: regenerated rewrite/multiplemerge's expected.tsv via its script.txt and diffed -- only the ms column (not compared by the protocol) differed. Co-Authored-By: Claude Fable 5 <[email protected]> review-fix: category is the authoritative pathological signal; enforce budget uniformly Runner now treats Category == "pathological" as authoritative; the manifest's redundant "pathological" boolean (now nullable, to detect absence) FAILs the fixture with a manifest-error message if present and disagreeing. Wall-clock budgets are enforced whenever Budget is present, not just for pathological fixtures. AdapterEngine uses the fixture's budget as its own process timeout when set (instead of the 5-minute default); Runner additionally wraps engine.Run in a Task.Run + Wait(budget + margin) watchdog for engines with no external process to kill (self-check) -- a runaway thread left behind on expiry is an acceptable trade in this short-lived CLI. Also hoisted the single expected.tsv read above the try block so a missing file can't throw from inside a catch. Verified: default run still 40/40, --include-pathological 41/41 (~2s vs 15000ms budget); manually flipping the manifest's pathological boolean FAILs with a manifest-error; manually shrinking the budget triggers both the watchdog-timeout path and the post-hoc over-budget path correctly (fixture restored after each test, working tree clean). Co-Authored-By: Claude Fable 5 <[email protected]> review-fix: mechanically enforce manifest.requires against grammar.xml PROTOCOL.md section 5 promised requires is mechanically derived from a fixture's grammar.xml, but nothing checked that. Add RequiresDerivation.Derive (scans grammar.xml for PhonologicalRule/MetathesisRule elements by LocalName) and have Runner validate every fixture's manifest.requires against it during the run, FAILing with a manifest-error message on mismatch. Update PROTOCOL.md/README.md to say the harness enforces this. Verified: full self-check run has 0 mismatches across all 41 fixtures (21 derive to ["phonology"], matching the documented count); --capabilities "" still skips exactly those 21; manually flipping one fixture's requires FAILs with a manifest-error, then the fixture is restored. Co-Authored-By: Claude Fable 5 <[email protected]> review-fix: propagate script-mode exit codes; stop discarding hc.dll stdout in wrapper Program.cs's script mode (-s file) ran DispatchCommand for every line but discarded its return value, and Main always returned 0 for script mode regardless of failures. Track the first nonzero DispatchCommand result and return it. Grepped src/tests/samples/.github for script-mode consumers relying on always-0 -- found none (CI only packs the Tool project, never runs it). hc-dotnet-wrapper.sh redirected hc.dll's stdout to /dev/null, which would also have swallowed load-error diagnostics AdapterEngine needs in its captured stderr. Redirect to stderr (>&2) instead. With the exit-code fix, the wrapper's `set -e` now genuinely propagates an hc.dll failure. Co-Authored-By: Claude Fable 5 <[email protected]> review-fix: dotnet csharpier format . Formats the 3 files edited during this review pass that CI's dotnet csharpier check . would otherwise flag (AdapterEngine.cs, Program.cs, BatchCommand.cs). Full repo check is now clean: Checked 709 files, 0 errors. Co-Authored-By: Claude Fable 5 <[email protected]> review-fix: update plan doc §9 stamp for expectCrash and requires-derivation changes Two acceptance-criteria bullets described mechanisms this review pass replaced: crash-matching was "the runner's dedicated crash-matching branch" inferring a crash from zero expected rows -- now it's the explicit manifest.expectCrash contract (FixtureManifest/EngineCrashException/Runner). requires-derivation was described as an external one-off cross-check script -- now it's RequiresDerivation.cs, enforced by Runner on every run. Updated both bullets to describe the current, harness-enforced mechanisms; counts (40/41, 19/21, 21 phonology-derived) are unchanged and still accurate. Co-Authored-By: Claude Fable 5 <[email protected]> review-fix: correct rewrite/expand provenance after regeneration manifest.json's oracle field still pointed at the buggy .worktrees/parse-opt @ ccf750e6 oracle that produced the original (wrong) frozen rows; update it to this branch's mainline oracle (.worktrees/conformance-framework @ f86895cc, the commit that actually regenerated expected.tsv). Update README.md's Generating command to the current mainline invocation, and rewrite the Resolved 2026-07-10 note with the true root cause: the original frozen rows were produced by the parse-opt oracle's LT-22613 Gate B/MaxAnalysisLength over-pruning bug (untraced mode only; fix commits 9a43d251/ab8471dc are not ancestors of ccf750e6), not by the untraced oracle "changing to agree" with the traced one. Mainline HermitCrab always accepted biiiibiiii. Co-Authored-By: Claude Fable 5 <[email protected]> review-fix: words.md -> words.yaml, correct coverage-report claim conformance/README.md and docs/conformance-framework-plan.md both documented a per-fixture words.md file that no fixture ships -- the 3 F3 fixtures use words.yaml (per the plan's own §10 open question, resolved to YAML), and the coverage report never reads any per-word file at all (CoverageReport.Build keys purely on manifest.json.constructs against constructs.txt). Rename every mention to words.yaml, drop the false "coverage report" claim, and note honestly that the 38 migrated fixtures don't have words.yaml yet (F4 backfill) while the 3 F3 fixtures do and all new fixtures must include it. Co-Authored-By: Claude Fable 5 <[email protected]> review-fix: remove dead rust/crates and rust/tools citations from PROTOCOL.md PROTOCOL.md cited rust/crates/hc-cli/src/main.rs and rust/tools/parse_compare.py as the origin of the batch contract and multiset-comparison rule -- paths that exist on no branch of this repo. PROTOCOL.md needs to be self-contained for third-party engine authors, so replace each citation with a plain statement that the convention originated in the Rust port's own CLI/comparison tooling, without a path. No normative content changed. Co-Authored-By: Claude Fable 5 <[email protected]> review-fix: resolve hc.dll default path relative to the wrapper's own location The optional default hc.dll path was a bare relative path (src/SIL.Machine.Morphology.HermitCrab.Tool/bin/Debug/net10.0/hc.dll), so it resolved against the caller's cwd rather than the repo root, breaking the wrapper when invoked from anywhere else. Resolve script_dir/repo_root from $0 instead, and try both Debug and Release net10.0 build outputs (first match wins) so either build configuration works without an explicit 5th arg. The argument contract is unchanged; an explicit hc.dll path still overrides the default outright. Verified: `cd <scratch-dir> && bash <repo>/conformance/adapters/hc-dotnet-wrapper.sh batch <abs grammar.xml> <abs words.txt> <abs out.tsv>` from a cwd outside the repo -> exit 0, 3-row TSV written. Co-Authored-By: Claude Fable 5 <[email protected]> review-fix: consistency sweep for stale words.md/oracle-narrative statements docs/conformance-framework-plan.md: one remaining words.md mention in the F3 phased-plan bullet (line ~280) missed by the earlier words.yaml rename; fix to words.yaml. The §9 acceptance-criteria bullet for rewrite/expand's stale expected.tsv still carried the same "the live oracle now finds the analysis" framing corrected in the fixture's own README (this review-fix pass) -- update it to state the actual root cause (parse-opt oracle's LT-22613 Gate B over-pruning bug at ccf750e6; mainline was never affected). docs/conformance-framework-implementation-notes.md: this doc is a historical planning record (written before expectCrash, category-authoritative pathological gating, words.yaml, and enforced requires-derivation existed) and is intentionally left unrewritten; add a one-line-per-item dated postscript at the top pointing at PROTOCOL.md/README.md for the current mechanics instead. Checked conformance/README.md and conformance/PROTOCOL.md for the same stale patterns (crash-inference-from-empty-expected, pathological boolean vs category, words.md, unenforced requires) -- both already carry the corrected language from this session's earlier code-review-fix commits (80795b7b, c726698e, c1ecf4cf), no further changes needed there. Co-Authored-By: Claude Fable 5 <[email protected]> docs: design spec for conformance suite v2 (typological synthetic languages) Restructure plan approved by John 2026-07-11: replace the 41-fixture mechanism taxonomy with 8 synthetic language grammars (plus an edge-cases micro-grammar set), words.yaml as single source of truth, generated coverage.csv, unchanged adapter protocol. Co-Authored-By: Claude Fable 5 <[email protected]> docs: gloss is per-parse and optional in the words.yaml schema A gloss describes one analysis: ambiguous words carry one per parse, expect_fail words carry none. Co-Authored-By: Claude Fable 5 <[email protected]> docs: per-parse rules list (trace-verified), non-parse and guess-stem coverage in schema - rules: required per parse, grammar.xml rule ids, verified against traced oracle applications; powers rules.csv and dead-rule detection - expect_fail words gain optional blocked_by attribution - guess: true marks LexicalGuess parses; signature rendering pinned in G1 - strict machine-parsable YAML conventions (schema-checked keys, no anchors/tags) Co-Authored-By: Claude Fable 5 <[email protected]> v2: add YAML fixture harness (discovery, self-check, adapter materialization, coverage CSVs, --propose) Phase G1 of the conformance-suite v2 restructure (docs/conformance-language-suite-plan.md). Adds a v2 fixture model (grammar.xml + words.yaml under conformance/languages/ and conformance/edge-cases/) alongside the existing, untouched v1 (manifest.json) model: - WordsYaml/WordsYamlLoader: strict YAML parser (YamlDotNet) for words.yaml's fixed key vocabulary -- unknown keys, anchors, aliases, merge keys, and custom tags are all hard errors. - FixtureV2: discovery under languages/ and edge-cases/, disjoint from v1's manifest.json-keyed discovery by construction. - GrammarRuleIndex: re-derives the grammar.xml `id` attribute for every rule kind from the XML directly, since none of AffixProcessRule/CompoundingRule/RewriteRule/MetathesisRule retain it at runtime (XmlLanguageLoader only uses it as a load-time dictionary key) -- this is what lets traced rule applications and words.yaml's "rules:"/"blocked_by" be compared on the same id space. - TraceRuleAttributor + RunnerV2.RunSelfCheck: per-word signature-set comparison against parses[].signature, plus traced verification of each parse's declared "rules:" list (documented, honest limits for phonological-rule attribution on ambiguous words, and for RealizationalAffixProcessRule, in the class doc comments). - FixtureMaterializer + RunnerV2.RunAdapter: materializes words.txt/expected.tsv into a temp v1 fixture shape and hands it to the existing, completely unmodified Runner/AdapterEngine for adapter mode -- the PROTOCOL.md wire contract does not change. - CoverageReportV2: emits conformance/coverage.csv and conformance/rules.csv, with dead-rule detection across every v2 grammar's declared rule ids. - ProposePatchWriter: --propose prints a would-be words.yaml patch on a signature mismatch; it only ever writes to the given TextWriter, never a file. - Program.cs: discovers and runs both v1 and v2 fixtures in one invocation, merging pass/fail into a single exit code. Co-Authored-By: Claude <[email protected]> v2: pin guess-stem signature rendering and document a signature-format quirk in PROTOCOL.md GuesserSignatureTests pins, with an executable test (since no CLI path in this codebase can exercise it -- see below), exactly how a Guesser/LexicalGuess parse renders in the 5-column batch signature: the guessed root's morph-chain component is the literal guessed surface substring itself (Morpher.LexicalGuess sets the throwaway LexEntry's Id to the matched text), with no distinguishing marker versus a real lexical entry. Also documents, and is the reason this needed a dedicated test rather than a `batch`-based conformance fixture: BatchCommand can never produce a guess-stem parse. SignatureFormat.ParseOneWord (what every `batch` invocation and the harness's own self-check both call for a plain word) hardcodes guessRoot: false, and no CLI command in SIL.Machine.Morphology.HermitCrab.Tool exposes the 3-argument overload at all. The v2 harness's self-check mode works around this per-word (RunnerV2.RunAllWords calls guessRoot: true whenever a word declares guess: true), but adapter mode structurally cannot -- recorded as a named, honest limitation rather than papered over. PROTOCOL.md gets a short new subsection documenting both findings, plus a related one surfaced while hand-deriving the suffixing-quechua pilot's signatures: multi-character character-definition segment representations (e.g. a digraph like "ch") always render individually parenthesized in a signature's shape half, regardless of ambiguity. Co-Authored-By: Claude <[email protected]> v2: author suffixing-quechua pilot language conformance/languages/suffixing-quechua/ (grammar.xml + words.yaml), the v2 pilot per docs/conformance-language-suite-plan.md section 3: invented language "Chaqru", Cuzco-Quechua inspired, strictly suffixing, requires: [] (zero phonological/metathesis rules -- the XAmple-eligible, morphotactics-pure grammar). Exercises: a 3-slot evidential-adjacent suffix chain (obligatory person slot + optional number/evidential slots), environment-conditioned allomorph selection done lexically (RequiredEnvironments on two disjoint-environment MorphologicalSubrules -- 1SG "-ni" post-consonant vs. "-y" post-vowel), disjunctive/free-fluctuating allomorphs (reportative evidential "-si"/"-shi", neither ever rejected in favor of the other), a MorphemeCoOccurrenceRule (conjectural evidential excluded from co-occurring with 2SG), obligatory-slot rejection, and a homophonous-root pair (eTupa/eTupa2, same shape "tupa") as the source of the required ambiguous 2-parse word. 13 words, 2 expect_fail (one blocked_by the co-occurrence rule, one the obligatory slot), zero dead rules. Hand-derived 5 of the 13 words' signatures before ever running the oracle (see the file's header comment for the derivations and reconciliation): 4 of 5 matched immediately; the 5th (walaknichik) did not, and the oracle was right -- multi-character segment representations ("ch") render individually parenthesized, a hand-derivation trap now also documented in PROTOCOL.md. Co-Authored-By: Claude <[email protected]> v2: convert loader-isactive and simultaneous-epenthesis-cascade edge cases conformance/edge-cases/loader-isactive/ and conformance/edge-cases/simultaneous-epenthesis-cascade/, converted from conformance/loader/n1-isactive and conformance/rewrite/simultaneous-epenthesis-cascade respectively (v1 originals left fully in place, untouched, until phase G4 -- see docs/conformance-language-suite-plan.md section 5). grammar.xml is copied verbatim in both cases; words.yaml carries the old manifest.json/README.md provenance text forward per-word. simultaneous-epenthesis-cascade sets expect_crash: true in front matter; its expect_crash semantics match v1 exactly, including FAILing (not passing) when the adapter binary can't start at all (a harness-level problem is never treated as a crash match, per PROTOCOL.md) -- verified via the reused, unmodified Runner/AdapterEngine adapter-mode path. Co-Authored-By: Claude <[email protected]> v2: commit generated coverage.csv/rules.csv for the current v2 fixture set Regenerated via --coverage-report over conformance/ (3 v2 fixtures). One expected, documented dead rule: edge-cases/simultaneous-epenthesis-cascade's prule4 -- the crash fixture's single word never completes parsing, so it structurally can never populate a parse's "rules:" list; this is not a gap in that fixture's authoring, it is what an expect_crash fixture's per-word data can express. The suffixing-quechua pilot itself has zero dead rules, as required. Co-Authored-By: Claude <[email protected]> v2: omit guess-only words from adapter materialization; add migration ledger Fable review of G1: BatchCommand can never produce guess parses (pinned in PROTOCOL.md), so materializing guess expectations into expected.tsv would falsely FAIL adapter mode on the first G2 guesser fixture. Guess- carrying words are now self-check-only, matching the documented contract. The ledger (docs/conformance-migration-ledger.md) assigns all 41 v1 fixtures to v2 destinations, marks the XAmple requires-[] floor (fusional- latin joins suffixing-quechua as a deliberately phonology-free grammar), and names the embed-or-edge-case fallbacks G4's parity proof keys on. Co-Authored-By: Claude Fable 5 <[email protected]> v2: author agglutinative-turkic pilot grammar (Kirenti) Embeds allomorphy/discontinuous-env, cross-cutting/disjunctive-tense- simultaneous-epenthesis, negative/obligatory-tense-slot, rewrite/disjunctive, rewrite/longdistance, rewrite/merge, rewrite/multiplemerge, rewrite/quantifier per docs/conformance-migration-ledger.md. Self-check green (19 words). Found and documented two engine/harness constraints during authoring (see words.yaml header): (1) a phonological subrule that changes a segment's identity is analysis-unreachable in self-check unless its Lhs/Rhs are FeatureNaturalClass over a fully-specified PhonologicalFeatureSystem, not a bare SegmentNaturalClass/literal Segment; (2) self-check's traced "rules:" verification records every phonological rule in a stratum as "applied" to every word regardless of whether it matched anything, so multi-rule grammars must declare the full phonological rule set on every parse. Co-Authored-By: Claude Fable 5 <[email protected]> v2: author fusional-latin grammar (Vetruna), requires: [] verified Embeds affix-shapes/circumfix, affix-shapes/infix, compounding/nonhead-not-root, compounding/prefix-commute, mpr-groups/output-overwrite, mpr-groups/required-all, realizational/family-blocking, realizational/realizational-rule, realizational/stem-name per docs/conformance-migration-ledger.md, plus ablaut as sole exponent (ModifyFromInput). Self-check green (49 words), requires: [] mechanically confirmed (zero PhonologicalRule/MetathesisRule elements). Six engine/authoring findings from this grammar (documented in words.yaml's header and inline in grammar.xml): compounding join markers must be a BoundaryDefinition, not a plain segment (zero parses otherwise); prefix morph-chain order is left-to-right surface position; unrelated suffix/POS/ ablaut-target collisions across grammar sections need explicit scoping (extra POS categories); and mpr-groups/output-overwrite (needs morphologicalRuleOrder linear) conflicts with compounding/prefix-commute's head recursion (needs unordered) within one stratum, resolved by splitting into two Strata. Co-Authored-By: Claude Fable 5 <[email protected]> v2: close mrPrefixBe1 dead-rule gap; regenerate coverage.csv/rules.csv Added a standalone "benox" word to fusional-latin (mrPrefixBe1 in isolation, proving the prefix rule itself is correct so panbenox's rejection is purely about the compounding non-head-must-be-a-root constraint). Zero dead rules now in both agglutinative-turkic and fusional-latin; the one remaining dead rule (edge-cases/simultaneous-epenthesis-cascade's prule4) predates this work and is out of scope. Co-Authored-By: Claude Fable 5 <[email protected]> v2: record G2a cross-cutting findings in the migration ledger Notes for future language authors (templatic-semitic, bantu-verbal, austronesian-phase, polysynthetic-inuit, prefixal-athabaskan): compounding's "+" must be a BoundaryDefinition; coexisting CompoundingRules need disjoint POS scoping; changing phonological subrules/ModifyFromInput need a fully- specified feature system; MPR-groups needs linear order which can conflict with other constructs needing unordered (split into two Strata); traced rules: verification can't distinguish "fired with effect" from "stratum pass merely ran" for multi-phonological-rule grammars. Co-Authored-By: Claude Fable 5 <[email protected]> v2: attribute only actually-applied phonological rules in traced rules verification TraceRuleAttributor counted every PhonologicalRuleSynthesis trace node, but TraceManager emits those for both applied (FailureReason.None) and not-applied (FailureReason set) attempts -- so in any grammar with >=2 phonological rules, every rule id appeared in every word's traced set and per-word attribution was meaningless. Filter to FailureReason.None nodes; narrow agglutinative-turkic's rules: lists to the truthful per-word sets (they match the header's hand derivations exactly); update the ledger finding from "workaround" to "fixed". Verified: self-check 40/40 v1 + 5/5 v2 green; corrupted-rules negative test FAILs (duy: declared [prVowelMerge,prGradation] != traced [prVowelMerge]); HC tests 67/67; rules.csv regenerated (sole dead rule remains pre-existing prule4 in the expect_crash edge case). Co-Authored-By: Claude Fable 5 <[email protected]> v2: attribute RealizationalAffixProcessRule firings via the synthesis trace walk Realizational rules are excluded from Word.MorphologicalRules by the engine (Word.cs), so they were unattributable and had to be omitted from successful parses'' rules: lists -- exactly the wrong gap for bantu-verbal, whose core constructs are realizational. But SynthesisRealizationalAffixProcessRule DOES report successful applications through ITraceManager.MorphologicalRuleApplied, so the same FailureReason.None-filtered trace walk that attributes phonological rules now also collects realizational rules (same word-level union caveat for ambiguous words). fusional-latin''s ferid now truthfully declares [rrPast]. Verified: self-check 40/40 v1 + 5/5 v2; HC tests 67/67; csharpier clean; rules.csv regenerated (rrPast now attributed on ferid, not just !feresid). Co-Authored-By: Claude Fable 5 <[email protected]> v2: author templatic-semitic (Zemreth) grammar + words.yaml Embeds v1 rewrite/expand, realizational/stem-name, rewrite/simultaneous-epenthesis, rewrite/simultaneous-feeding(-control-iterative), rewrite/word-initial-epenthesis. Carries InsertSimpleContext/ModifyFromInput (zero coverage today), OCP root co-occurrence, RewriteRule Simultaneous, epenthesis/prothesis. Self-check green. Co-Authored-By: Claude Opus 4.8 <[email protected]> v2: author bantu-verbal (Velamba) grammar + words.yaml Embeds v1 mpr-groups/output-overwrite, mpr-groups/required-all, realizational/realizational-rule, rewrite/required-pos-subrule. Carries realizational rules (primary home), MPR features/groups, verbal reduplication (ReduplicationHint, zero coverage today), extension-slot ordering, nasal-prefix mutation. Self-check green; documents a genuine AffixTemplate analysis-side ordering-permissiveness finding (see andikilisha's note). Co-Authored-By: Claude Opus 4.8 <[email protected]> v2: regenerate coverage.csv/rules.csv for templatic-semitic + bantu-verbal Zero new dead rules; every rule in both new grammars is exercised by at least one word. Only pre-existing dead rule remains: prule4 in edge-cases/simultaneous-epenthesis-cascade. Co-Authored-By: Claude Opus 4.8 <[email protected]> v2: author austronesian-phase grammar.xml (infix/circumfix/redup/truncate/metathesis/noncontiguous) Co-Authored-By: Claude Opus 4.8 <[email protected]> v2: austronesian-phase words.yaml green (metathesis x3, infix -um-, circumfix, redup, truncate, noncontiguous) Co-Authored-By: Claude Opus 4.8 <[email protected]> v2: author polysynthetic-inuit grammar.xml (stratum ordering, compounding x2, guesser, seam phonology x3) Co-Authored-By: Claude Opus 4.8 <[email protected]> v2: polysynthetic-inuit words.yaml green (stratum ordering, compounding x2, guesser, seam phonology x3) Guesser required isolating the seam-rewrite rules into their own deeper stratum: with the rewrite rules in the same stratum as the guess pattern, analysis-direction flooding defeats LexicalGuess (returns nothing). Three-stratum grammar (Phonology deepest / Derivation / Inflection). Co-Authored-By: Claude Opus 4.8 <[email protected]> v2: exercise prMDC2 (add matu/mau), drop misplaced provenance, regenerate CSVs Adds a posMDC root 'matu' -> 'mau' so the P6 t-deletion rule prMDC2 (inert on the buuubuuu composition, where only its analysis-direction reversal matters) is exercised by a word, clearing the last new dead rule. Only prule4 (pre-existing) remains dead. Removes a mis-pasted compounding provenance from the stratum-ordering word nunaliqvuq. Co-Authored-By: Claude Opus 4.8 <[email protected]> v2: restore 'W9.1 probe, ' prefix on austronesian-phase affix-shapes provenance The four affix-shapes provenance strings (Circumfix/Infix/NonContiguous/ Truncate) were authored without the "W9.1 probe, " prefix the v1 manifests carry, which would fail the G4 provenance-parity grep (every v1 manifest.json.provenance must appear verbatim in a v2 provenance field). Co-Authored-By: Claude Opus 4.8 <[email protected]> v2: migrate pathological/deep-optional-affix-nesting to edge-cases (budget_ms) Generates all 924 parses:// entries for the C(12,6) combinatorial-blowup word mechanically from v1's own oracle-verified expected.tsv, then self-check-verifies every declared (signature, rules) pair against the live oracle. requires: [] confirmed (no PhonologicalRule/MetathesisRule element in the grammar). v2: migrate loader/n2-default-symbol and loader/n3-pattern-shapes to edge-cases XML-loader-semantics probes, provenance carried verbatim. Both requires: []/[phonology] match the v1 fixtures exactly; grammar.xml files are byte-identical copies. v2: migrate allomorphy/disjunctive-recheck and strrep-identity to edge-cases Both v1 fixtures deliberately declare no <MorphemeId> on their MorphologicalRule elements (adding one would change the "+|wokta"-style signatures they pin), which exposed a genuine GrammarRuleIndex gap: ResolveMorphologicalRuleId could not attribute a fired rule back to its grammar.xml id at all in that case, forcing a dishonest "rules: []" and permanent dead-rule flags on ruleT/ruleD/rulePfx/ruleObj. Fix: GrammarRuleIndex now registers a rule's <Name> text in the same fallback dict CompoundingRule/PhonologicalRule/MetathesisRule already use, but only when the rule has no <MorphemeId> at all -- conservative by construction, so all 8 existing language grammars (every rule has a MorphemeId) are byte-for-byte unaffected. Confirmed empirically before/after (see disjunctive-recheck/words.yaml header). v2: migrate affix-shapes/truncate to edge-cases/truncate-morphotactic (XAmple floor) requires: [] confirmed (no PhonologicalRule/MetathesisRule element) -- the phonology-free home for truncation coverage the ledger calls out, since austronesian-phase's own truncate word lives in a requires:[phonology] grammar. Same provenance string as austronesian-phase's mrTrunc word, carried deliberately (both pin the identical W9.1 construct). v2: author prefixal-athabaskan (Ketselan) language grammar Pure STRESS grammar (spec Sec3 row 7): a left-edge prefixal AffixTemplate with 6 position-class slots, exercising a fused mode+transitivity portmanteau, an adjacent-dependency disjunctive subject allomorph gated by lexically-preset root class, and a genuinely discontinuous slot dependency (object's admissibility gated by a feature set two non-adjacent slots inward, skipping the obligatory subject slot). requires: [] (phonology-free, a second XAmple-floor grammar alongside suffixing-quechua/fusional-latin). 10 hand-derived words authored before the oracle; 9/10 matched on the first self-check run. The tenth ("gahobishiyidkal", an order-swap test modeled on bantu-verbal's andikilisha finding) was hand-predicted ACCEPTED by the same fixpoint-retry mechanism -- the oracle actually REJECTS it. Traced the reason to AnalysisAffixTemplateRule.ApplySlots: bantu-verbal's swappable optional pair has nothing but optional slots between it and the root, but this template's swappable pair (advA/advB) has three obligatory slots in the way, which blocks the single-pass outer-to-inner analysis walk from ever revisiting the out-of-turn affix. Corrected to expect_fail with the mechanism documented in the word's own note -- G2c's finding is real but conditioned on slot shape, not universal. v2: regenerate coverage.csv/rules.csv for G2d's 7 new fixtures prefixal-athabaskan + 6 migrated edge-cases. Zero new dead rules -- only the pre-existing prule4 (edge-cases/simultaneous-epenthesis-cascade) remains. v2: add absolute construct-coverage check + migration parity proof (G2e) Adds the mechanical guard for the one irreversible migration step (deleting the v1 fixture tree), per docs/conformance-language-suite-plan.md G2e/G4: - CoverageReportV2 now tracks the set of constructs any v2 word exercises; --coverage-report reports absolute coverage against constructs.txt (every in-scope construct covered; Tracing out of scope by design). This is the v2-native replacement for the v1 CoverageReport zero-coverage rollup, which goes inert once the v1 tree is deleted. - conformance/parity-check.py: committed, re-runnable, exits nonzero on any violation. Checks the three ledger floors (provenance verbatim v1->v2, per-construct floor, XAmple requires:[] floor) plus the absolute check. Self-validates it discovered exactly 41 v1 fixtures. The proof currently reports 3 REAL violations: suffixing-quechua (G1 pilot, authored before provenance discipline) never carried the cooccurrence/ {allomorph-basic,morpheme-adjacency,and-semantics-pin} provenance strings and does not genuinely exercise their sub-behaviors. Fixed in the next commit; the v1 tree is deleted only once this proof is green. Co-Authored-By: Claude Opus 4.8 <[email protected]> v2: embed 3 missing v1 cooccurrence fixtures into suffixing-quechua grammar.xml Adds: AllomorphCoOccurrenceRule (eTaki/aKanta doublet, W6 allomorph-basic), 4 new MorphemeCoOccurrenceRule adjacency kinds (mrEvidAssum/mrEvidMir/ mrPerson1PlIncl/mrPerson2Pl, W6 morpheme-adjacency), and a second exclude rule on mrEvidConjectural (AND-not-OR semantics, LT-22156). words.yaml follow-up commit will add/annotate the exercising words. v2: add 12 new words + provenance for the 3 embedded cooccurrence fixtures allomorph-basic (eTaki/aKanta doublet): takiy, kantay, takincha, kantancha. morpheme-adjacency (4 new adjacency-kind rules): walaknitan, walaknichiktan, walakniwas, walaknichikwas, siputukumi, siputukuchikmi, sipulupami, sipulupachikmi. and-semantics-pin: provenance + note update on the existing walaknkicha/sipuncha pair (now the AND-not-OR pin, given grammar.xml's new second exclude rule on mrEvidConjectural). v2: regenerate coverage.csv/rules.csv for suffixing-quechua's new rules No new dead rules -- only the pre-existing edge-cases/simultaneous-epenthesis- cascade prule4 remains. v2: close somewhereTo* no-op gap; move allomorph provenance to the blocked word Adds walakntan/siputuku: expect_fail words where the somewhereToLeft/ somewhereToRight target is entirely absent (not just non-adjacent), so a no-op "somewhere*" implementation (always-satisfied) would now be caught -- the prior word set only varied adjacency while keeping the target present. Moves "W6, AllomorphCoOccurrenceRule granularity" provenance from takiy (unblocked, no rule attached) to kantancha (the word the rule actually blocks), matching where and-semantics-pin's provenance lives (on the pin, not an unrelated positive word). v2: record the passing G2e parity proof in the migration ledger The parity proof (conformance/parity-check.py) is green: all three floors (provenance verbatim, per-construct, XAmple requires:[]) plus absolute construct coverage pass, with exactly 41 v1 fixtures discovered. This is the gate for deleting the v1 tree, which follows in the next commit. Co-Authored-By: Claude Opus 4.8 <[email protected]> v2: delete the v1 fixture tree (41 fixtures) — parity proof green The G2e parity proof (conformance/parity-check.py, recorded in the ledger) is green: all three migration floors plus absolute construct coverage pass, so every v1 construct, requires:[] home, and provenance string is preserved in the v2 languages/ + edge-cases/ suite. Removes the 12 v1 category directories (affix-shapes, allomorphy, compounding, cooccurrence, cross-cutting, loader, metathesis, mpr-groups, negative, pathological, realizational, rewrite) — 247 files. The migration ledger (docs/conformance-migration-ledger.md) is retained permanently as the provenance bridge. HISTORY-MATRIX.md and the v1 plan are archived in G3. Co-Authored-By: Claude Opus 4.8 <[email protected]> G3: rewrite conformance/README.md for v2 shape Old README described the v1 fixture-per-mechanism layout (words.txt/expected.tsv/ manifest.json per fixture); rewrite for the v2 shape: languages/ + edge-cases/, two files each, words.yaml as ground truth. Covers the eight synthetic languages and what each simulates, coverage philosophy (constructs.txt, verified rules: dimension, dead-rule detection, 19/19 in-scope constructs), how to run the harness, and the growth policy. 101 lines (target <=120). G3: trim PROTOCOL.md prose and fix stale v1 references Contract stays byte-identical in meaning (CLI batch signature, TSV format, signature algorithm, guess-stem rendering pin, comparison semantics, capability profiles, per-engine grammar representation, hc.dll wrapper aside): only redundant/verbose prose was cut, plus several stale artifacts fixed for accuracy now that v1 is gone: - manifest.json -> words.yaml front matter (requires, expect_crash location) - "negative fixture category" -> expect_fail (v2 dissolved the category) - stale fixture-path example rewrite/simultaneous-epenthesis-cascade -> edge-cases/simultaneous-epenthesis-cascade - pre-existing wrong internal cross-references (section 5->6, section 6->7, section 3->4) corrected - "the design doc" pointers resolved to docs/archive/conformance-framework-plan.md now that it has moved 272 -> 263 lines. G3: archive superseded v1 planning docs under docs/archive/ git mv (history preserved) conformance-framework-plan.md (v1 design plan, superseded by conformance-language-suite-plan.md), HISTORY-MATRIX.md (live content already in the migration ledger/provenance fields), and conformance-framework-implementation-notes.md (v1 F0-F3 scoping note; fully v1-specific -- 38-fixture migration, manifest.json/script.txt backfill, rust-branch path fixes -- and its own postscript already defers to PROTOCOL.md/README.md for current behavior). Add docs/archive/README.md noting these are historical, kept for provenance. Fix the one broken link this surfaced: conformance-language-suite-plan.md's "Supersedes ... docs/conformance-framework-plan.md" now points at the new docs/archive/ path. G3: update constructs.txt header comment for v2 Constructs are now drawn from each words.yaml's per-parse/per-word "exercises:" field (languages/ and edge-cases/), not manifest.json's "constructs" array; the coverage report cross-references against this list and flags both zero-coverage constructs and dead rules. Fixed the reference to the archived plan doc's path. No construct lines changed. v2: fix dangling PROTOCOL.md reference in GrammarRuleIndex co-occurrence doc The class comment cited "conformance/PROTOCOL.md's v2 addendum" for the co-occurrence pseudo-id convention, but that addendum never existed in PROTOCOL.md (a pre-existing over-promise, not removed by the G3 trim). The convention is coverage/dead-rule bookkeeping internal to the harness, not part of PROTOCOL.md's wire adapter contract, and is fully documented in this class comment itself -- point there instead. Co-Authored-By: Claude Opus 4.8 <[email protected]> v2: add expect_skip status so adapter mode distinguishes SKIPPED from ok Adapter mode (the materializer -> subprocess -> diff path, which self-check never exercises) surfaced a real gap: loader-pattern-shapes' "bit" word uses an undeclared segment, so the oracle SKIPS it (InvalidShapeException) with adapter-contract status "SKIPPED", but FixtureMaterializer hardcoded status "ok" for every non-crash word -> spurious status mismatch. Self-check hid it by collapsing InvalidShapeException to an empty parse set. New words.yaml key `expect_skip: true` (mutually exclusive with expect_fail) pins the SKIPPED status as authored ground truth. Both run modes now honor the distinction and stay consistent: self-check verifies an InvalidShapeException actually fired (and that an expect_fail word was NOT skipped), and FixtureMaterializer materializes status "SKIPPED". Converted "bit" to expect_skip. Verified: self-check 16/16 and adapter mode 16/16 both green; a negative test (bit mislabeled expect_fail) FAILs as intended. Co-Authored-By: Claude Opus 4.8 <[email protected]>
johnml1135
force-pushed
the
conformance-framework
branch
from
July 12, 2026 03:42
8b75e1b to
b04abfe
Compare
The session that added the mprRRealTest MorphologicalPhonologicalRuleFeature appended it after the FeatureGroup elements, violating the DTD's element ordering and crashing the loader. Reordered it before the groups. Also, snG5's StemName Region carried an AssignedHeadFeatures constraint (featTense=symPres) that nothing in the daleh/daheh derivation ever assigns, so RootAllomorph's StemName.IsRequiredMatch check failed root selection before the intended SynthesisAffixProcessRule.RequiredStemName check could even run. Dropped the head-feature constraint so the region only requires posV7, restoring the intended failure/success split (daleh parses, daheh does not). Self-check 15/15, coverage 24/24 in-scope constructs, HC tests 67/67, csharpier clean.
Pins the PanGloss hc-foma (alternative FST) engine's Latin-diacritic zero-parse bug: words containing precomposed Latin diacritics (é, ö, ñ, î) returned zero parses under that engine while HermitCrab and Cyrillic text parsed fine. The bug is engine-internal (Unicode NFD-decomposition interacting with hc-foma's own byte/token handling), not a phenomenon any of the 8 typological languages can host naturalistically (they use plain-ASCII synthetic lexemes by design), so per conformance/README.md's growth policy this lands as an edge-case (option 3), not a language word. Four roots (año/café/göl/kelî) place each diacritic at a different position (medial vs. stem-final); two optional suffix rules (rPl unconditional, rDim gated by a LeftEnvironment NaturalClass check requiring a diacritic-final stem) exercise both bare and affixed forms, plus one SKIPPED pin for an undeclared diacritic (ü). All 13 words' signatures are hand-derived and self-check-verified against the C# oracle (dotnet run --project src/SIL.Machine.Morphology.HermitCrab.Conformance -- --fixtures conformance --include-pathological: 17/17 fixtures pass). coverage.csv/rules.csv regenerated via --coverage-report (24/24 in-scope constructs covered, 0 new dead rules); parity-check.py still passes. Co-Authored-By: Claude Sonnet 5 <[email protected]>
…amily Renames every language-family-named fixture directory (and its grammar.xml <Name>, comments, and words.yaml language/inspired_by/sources metadata) to describe the construct or pathology it composes instead: agglutinative-turkic -> suffixing-vowel-harmony templatic-semitic -> templatic-root-modification fusional-latin -> fusional-realizational-morphology polysynthetic-inuit -> polysynthetic-stratal-derivation-chain bantu-verbal -> suffixing-extension-slot-ordering prefixal-athabaskan -> prefixal-discontinuous-slot-dependency suffixing-quechua -> suffixing-evidential-adjacency-chain austronesian-phase -> metathesis-phase-isolation edge-cases/latin-diacritic-segments -> edge-cases/diacritic-segments Typology adjectives (suffixing/prefixal/templatic/polysynthetic/fusional) are kept; language/family nouns (Turkish, Arabic, Latin, Swahili, Navajo, Quechua, Tagalog, Bantu, Inuit, Hebrew, Semitic, ...) are removed from directory names, the grammar's own <Name>, comments, and words.yaml `inspired_by`/`sources` metadata. All fixture semantics (lexemes, rules, signatures, exercises) are unchanged -- this is a pure rename/relabel. `edge-cases/latin-diacritic-segments` keeps "Latin" only as a script reference in its own content (it is the writing system under test, not a family noun), per the same convention as LatinTokenizer in machine/src. coverage.csv and rules.csv regenerated via the conformance driver's own --coverage-report (same 24/24 construct coverage, same 1 dead rule, same per-word signatures -- diff confirms only the language-name column moved). conformance/README.md and conformance/PROTOCOL.md cross-references updated to match. Verified with `dotnet run ... --fixtures conformance` (self-check green) after every edit.
conformance_coverage.rs's mapping (machine-consuming PanGloss repo) found 4 CharacteristicKind values -- rewrite direction (LeftToRightRewrite/ RightToLeftRewrite), phonological subrule-level MPR/POS gating (SubruleGating), and multi-CharacterDefinitionTable stratification (MultiTable) -- with no constructs.txt row distinctly tagging them, so no fixture could ever be written to exercise them by name. The existing "RewriteRule Iterative/Simultaneous" rows tag application-order, not directionality; "MPR features/groups" is morphological MprGroup territory, not phonological subrule gating; the two existing CharacterDefinitionTable rows are about segmentation/pattern constructs within one table, not about multiple tables disagreeing across strata. Add one row per characteristic so each can be evidenced independently.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Engine-agnostic morphological-parser conformance suite (v2: typologically-selected languages)
An engine-agnostic conformance test suite for what a morphological/phonological parser must get right — a portable "all-language parser oracle". The C# HermitCrab engine is the founding oracle that generated ground truth; any engine implementing
conformance/PROTOCOL.md's adapter contract can be validated against it.This PR began as the F0–F3 fixture-per-mechanism suite and has been restructured into v2: a small set of dense, typologically-selected synthetic languages, replacing the 41 hand-minimized single-mechanism fixtures. Full design + rationale:
docs/conformance-language-suite-plan.md.Why restructure
The F0–F3 suite worked but (1) sprawled to 252 files (up to 7 per fixture, each word's story smeared across four), (2) made isolation the default when C#'s real bugs cluster in interactions, and (3) let coverage follow engine class-names rather than language phenomena — six constructs sat at zero coverage because no bug happened to hit them.
The v2 shape
conformance/languages/) + 8 edge-cases (conformance/edge-cases/), exactly two files each (grammar.xml+words.yaml). 252 → 40 files.words.yamlis the single ground truth: per-wordnote+parses(signature/gloss/rules/exercises) orexpect_fail/expect_skip, plusprovenance,guess,budget_ms/expect_crash.inspired_by/sourcesname only the phenomenon class each mechanism simulates (e.g. "Swahili/Sena-style reduplication"); no real-language grammar (Sena, Indonesian, Amharic, or any other) is included in this suite.Coverage: nothing regressed, five zero-coverage constructs closed
Tracingis out of scope by design — never inexpected.tsv's domain). The five previously-zero constructs now have homes: reduplication (ReduplicationHint),ModifyFromInput/InsertSimpleContext,CopyFromInput/InsertSegments, stratum ordering, andGuesser/LexicalGuess.rules:field is a verified coverage dimension: self-check runs the oracle with tracing on and FAILs if the traced rule applications diverge from the declared list.prule4in theexpect_crashcascade edge-case, which is un-exercisable by design (the engine crashes withInfiniteLoopExceptionbefore any parse completes, so no word can ever attribute it).The migration was mechanically proven before anything was deleted
conformance/parity-check.py(committed, re-runnable, exits nonzero on any violation) enforces the three migration floors + an absolute construct check, and self-validates that it discovered exactly 41 v1 fixtures. It gated the deletion of the v1 tree:It caught a real gap during authoring (the G1 pilot never carried three
cooccurrence/*provenance strings) which was fixed before green.docs/conformance-migration-ledger.mdis retained permanently as the v1→v2 provenance bridge; every v1manifest.json.provenanceappears verbatim in some v2provenance:field.Both run modes verified
words.txt/expected.tsv→ subprocess → diff, viahc-dotnet-wrapper.sh): 16/16 PASS. Running it surfaced (and this PR fixes) a materializer gap — anInvalidShapeException-skipped word was materialized as statusokinstead ofSKIPPED; the newexpect_skipkey pins that status and both modes now verify it consistently.Harness additions
verified traced-
rules:self-check;--coverage-reportnow reports absolute construct coverage againstconstructs.txt(v2-native, survives the v1 deletion);--propose(prints awords.yamlpatch on signature mismatch, never writes);expect_skipstatus;conformance/parity-check.py. The adapter wire contract inPROTOCOL.mdis unchanged.Docs
One
conformance/README.md(≤120 lines);PROTOCOL.mdtrimmed (contract untouched); the v1 plan, HISTORY-MATRIX, and implementation notes archived underdocs/archive/.🤖 Generated with Claude Code
This change is