FidelityTTT is a research prototype for scientific search when experiments come in several levels of cost and reliability.
The system learns two decisions together:
- Which hypothesis should be tested next
- Which evaluator is worth paying for
Cheap proxies guide exploration. Medium evaluators estimate and correct proxy bias. Reference evaluators decide what counts as a verified discovery. Policy updates are weighted by reliability, calibration, reproducibility, validity, and proxy exploitation risk.
The default path is deterministic, CPU safe, and does not require an API key.
- Five executable multi-fidelity domains
- A cost-aware Bayesian controller
- Bootstrap proxy-bias calibration
- Confidence-weighted proposal updates
- Confirmation and replication scheduling
- SQLite and JSONL experiment memory
- Honest high-fidelity metrics
- Eleven budget-matched baselines
- A local Streamlit dashboard
- A hosted interactive browser lab
- Replayable run artifacts and tests
FidelityTTT is not a resource scheduler with a new label. It models bias between fidelities, uses that calibration inside the acquisition rule, gates proposal updates, and separates proxy winners from verified discoveries.
Create an environment and install the package:
python -m venv .venv
# Linux and macOS
source .venv/bin/activate
# Windows PowerShell
.venv\Scripts\Activate.ps1
pip install -e .Run the seeded synthetic benchmark:
python -m fidelityttt.run --config configs/synthetic_small.yamlLaunch the local research dashboard:
python -m fidelityttt.app --config configs/live_demo.yamlRun symbolic regression:
python -m fidelityttt.run --config configs/symbolic_regression.yamlEvaluate the latest saved run:
python -m fidelityttt.evaluate --run-dir runs/latestRun the test suite:
pytestflowchart LR
A["Proposal policy"] --> B["Hypothesis memory"]
B --> C["Bayesian controller"]
D["Proxy-bias model"] --> C
E["Budget state"] --> C
C --> F["Low proxy"]
C --> G["Medium evaluator"]
C --> H["Reference evaluator"]
F --> I["Experiment log"]
G --> I
H --> I
I --> D
I --> J["Confidence gate"]
J --> A
I --> K["Confirmation scheduler"]
K --> C
I --> L["Verified metrics"]
The controller scores a hypothesis and fidelity pair with:
acquisition =
expected reference improvement
+ information gain about proxy bias
+ confirmation value
+ expected proposal update value
- evaluation cost
- proxy exploitation risk
- expected latency
The controller never reads a reference label before it pays for the corresponding reference evaluation.
An experiment contributes to the proposal policy through:
update weight =
fidelity reliability
x calibration confidence
x reproducibility confidence
x validity confidence
x non-exploitation confidence
Low-fidelity outliers receive small weights. Invalid results receive zero positive weight. Suspicious proxy winners are useful as negative evidence. An update is skipped when its calibrated signal is weak or harmful.
The default implementation updates a versioned heuristic proposal table. The adapter boundary supports learned policies without making the CPU demo depend on GPU packages.
| Domain | Low fidelity | Medium fidelity | Reference fidelity | Main proxy trap |
|---|---|---|---|---|
| Synthetic discovery | Biased scalar proxy | Partially corrected simulation | Hidden objective | Attractive decoy region |
| Symbolic regression | Small central subset | Broader noisy subset | Full held-out interval | Narrow local fit |
| Algorithm design | Tiny public instances | Random stress suite | Hidden adversarial suite | Greedy shortcut |
| Numerical kernel | Static estimate | Short shape benchmark | Full correctness benchmark | Fast but brittle configuration |
| Materials surrogate | Cheap surrogate | Strong surrogate | Reference model | Out-of-distribution artifact |
Every domain implements the same interface:
class FidelityDomain:
def initial_hypotheses(self) -> list[Hypothesis]: ...
def mutate(self, hypothesis: Hypothesis, seed: int) -> Hypothesis: ...
def random_hypothesis(self, seed: int) -> Hypothesis: ...
def fidelity_levels(self) -> list[FidelityLevel]: ...
def evaluate(self, hypothesis, fidelity_id, seed) -> ExperimentResult: ...
def oracle_best_score(self) -> float | None: ...Each run writes a self-contained directory:
runs/<timestamp>/
config.yaml
run.sqlite
hypotheses.jsonl
fidelity_levels.jsonl
results.jsonl
decisions.jsonl
policy_updates.jsonl
proxy_bias_snapshots.jsonl
metrics.json
dashboard_snapshot.html
Every experiment records its seed, fidelity, cost, latency, validity, raw metrics, and replicate index. Every controller decision records its acquisition terms and a plain-language explanation.
runs/latest.json points to the latest run. The evaluator accepts runs/latest as an alias.
The repository includes:
- Random search
- Frozen evolutionary search
- Cheap-proxy greedy search
- Equal-budget high-fidelity search
- Hyperband
- Successive halving
- Multi-fidelity Bayesian optimization
- Low-fidelity test-time training
- High-fidelity test-time training
- Confirm-all top-k
- An oracle controller for diagnostic upper bounds
Baselines receive the same total cost budget and the same configured limit on reference evaluations.
The checked synthetic configuration uses seed 11, total cost 180, and a limit of 12 reference evaluations.
| Method | Best verified reward | Reference evaluations |
|---|---|---|
| FidelityTTT | 0.9623 | 11 |
| MultiFidelityBO | 0.9568 | 8 |
| CheapProxyGreedy | 0.9553 | 12 |
| SingleFidelityTTTHigh | 0.9365 | 12 |
This table is a deterministic acceptance test, not a publication claim. A paper result requires the multi-seed protocol, held-out bias families, confidence intervals, and ablations in docs/EXPERIMENTS.md.
Primary metrics:
- Best high-fidelity verified reward
- Best verified reward by total cost
- Simple regret
- Cost to reach a target
- Reference evaluations used
- Proxy exploitation rate
- Calibration error
- Confirmation precision and recall
- Reproducibility pass rate
- Proposal update harm
- Update acceptance rate
- Wall-clock time
Low-fidelity scores never count as discoveries.
fidelityttt/
domains/ executable benchmark domains
adapters/ optional learned-policy and GPU boundaries
app.py local Streamlit application
engine.py end-to-end discovery loop
controller.py cost-aware acquisition
proxy_bias.py bootstrap calibration model
policy_update.py confidence gate
confirmation.py escalation scheduler
reproducibility.py replicate analysis
baselines.py budget-matched comparisons
memory.py in-memory scientific state
storage.py SQLite and JSONL artifacts
metrics.py honest discovery metrics
configs/ runnable experiment definitions
tests/ unit, integration, replay, and acceptance tests
app/ hosted interactive lab
docs/ experimental and paper roadmap
The CPU path never imports GPU packages. Install the optional group when developing a learned proposal policy:
pip install -e ".[gpu]"The current QLoRA adapter is an explicit extension boundary. It checks dependencies and refuses silent fallback. A production implementation should add reward-weighted examples, held-out update validation, checkpoint rollback, and deterministic dataset snapshots.
- All benchmark noise comes from recorded seeds
- Domain evaluators are deterministic for a fixed seed
- Reference labels are unavailable to the controller until evaluated
- Costs and reference counts are enforced before execution
- SQLite mirrors append-only JSONL artifacts
- The HTML snapshot can be reviewed without the live app
- CI runs both the Python suite and the hosted application build
MIT