Heretic
Heretic Future and Project: Build an Auditable Model-Editing Lab
A capstone design for reproducible Heretic experiments with manifests, evaluation gates, residual visualizations, and responsible release.
What you will learn
- Explain the KL/refusal optimization objective
- Run a pinned model experiment
- Design broader safety evaluation
Before you start
- Basic Git and command-line usage
- Comfort reading a project README
You can explain the project, run its documented first step, and decide what to verify before adopting it.
Key takeaways
- An evidence manifest turns a model edit into a reproducible claim.
- Capability profiles should be explicit for each architecture and quantization.
- Interactive visuals are optional; accessible text remains canonical.
The project brief
Build a private experiment lab that accepts a model revision and prompt manifest, runs baseline and Heretic transforms, stores residual and optimization evidence, and emits a reviewable comparison package. The package should make it possible to reproduce a result without trusting a screenshot or a single score.
Start with one supported dense model and one local GPU. Add multimodal, MoE, quantized, and research-plot variants only as explicit capability profiles, because the README does not promise universal architecture support.
Evidence manifest
Record model and tokenizer revisions, license, dependency lock, hardware, configuration, prompt hashes, direction settings, Optuna budget, refusal/KL/capability results, output hashes, reviewer, and publication decision. Keep raw prompts and weights in controlled storage; publish sanitized fixtures and hashes.
Use Markdown and local SVG as the canonical report. A Three.js view can optionally explore layers and residual trajectories, but it must not hide the textual metrics, warnings, or accessibility path.
Roadmap and gates
Phase one is trace-only reproducibility; phase two adds advisory quality and safety checks; phase three can enforce release gates for license, provenance, regression, and human sign-off. Future research can explore better objectives, architecture coverage, and interpretability while retaining rollback and quarantine controls.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Define model, prompt, run, metric, artifact, and review fields.
- 2
Add offline fixtures and baseline/candidate parity checks.
- 3
Gate publication on license, provenance, safety, and capability review.
- 4
Promote trace-only checks to advisory and enforced release policies.
Copy-ready example
run:
model_revision: <sha>
prompt_manifest_sha256: <hash>
config: config.default.toml
hardware: <gpu-pytorch-cuda>
metrics: { refusal: <n/total>, kl: <value> }
checkpoint_sha256: <hash>
reviewer: <id>
release: holdFrequently asked questions
Should the lab store model weights in Git?
No. Keep large or sensitive artifacts in controlled storage and commit only sanitized fixtures, hashes, and manifests.
Is Three.js required for the project?
No. Add it only if a layer/residual exploration view helps; preserve accessible text, SVG, and downloadable evidence.
Sources
- Heretic README (captured 2026-08-31)Source checked 2026-08-31
- Heretic repositorySource checked 2026-08-31