Heretic
Heretic Comparison and Selection: Abliteration Tool, Fine-Tuning, or Evaluation Lab?
Compare Heretic with manual abliteration, fine-tuning, refusal editing, and hosted model choices by evidence, control, cost, and risk.
What you will learn
- Explain the KL/refusal optimization objective
- Run a pinned model experiment
- Design broader safety evaluation
Before you start
- Basic Git and command-line usage
- Comfort reading a project README
You can explain the project, run its documented first step, and decide what to verify before adopting it.
Key takeaways
- Select a method based on the research question and governance burden.
- Use an equal prompt and capability fixture for comparisons.
- Fewer refusals alone do not establish a better model.
Compare the actual decision
Heretic is a specialized, automatic directional-ablation workflow. Manual abliteration offers more hand control; fine-tuning changes a broader training surface; hosted or already-decensored models reduce operator work but reduce transformation control. The right choice depends on the research question, license, privacy, and review capacity.
Use the same model family, prompt manifest, capability suite, and publication policy for a fair bake-off. Compare refusal behavior, KL or other fidelity measures, factuality, coding/math tasks, calibration, latency, cost, and human judgments.
Make trade-offs explicit
Heretic's strengths are automation, flexible kernels, and reproducible CLI runs. Its boundaries include supported architectures, GPU/CPU cost, metric sensitivity, and the responsibility to evaluate transformed artifacts. A hosted option may win on operations; a manual method may win when a researcher needs fine-grained hypotheses.
Do not equate fewer refusals with better alignment or usefulness. Document what each candidate cannot answer, how it handles unsafe requests, and how quickly a team can revoke or replace it.
A selection record should name the owner, evaluation window, release policy, and rollback trigger so a later team can understand why the method was chosen.
A reviewable next step
Use the pinned source evidence, record versions and assumptions, and ask a qualified reviewer to validate the result before adopting it.
Use the pinned source evidence, record versions and assumptions, and ask a qualified reviewer to validate the result before adopting it. Keep the fixture, output, and decision together so another developer can reproduce the conclusion.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Define behavior, fidelity, cost, license, and risk criteria.
- 2
Run equal baseline and candidate evaluations.
- 3
Add human review and operational tests.
- 4
Record decision, limits, and reevaluation triggers.
Copy-ready example
question -> methods
model/prompt fixture -> refusal + fidelity + capability
ops + license + review -> decision + exit planFrequently asked questions
Is Heretic a replacement for fine-tuning?
No. It targets a different intervention surface; choose based on whether directional weight edits answer your research question.
How do I compare an existing decensored checkpoint?
Use the same model family, prompts, software context where possible, and report provenance plus human evaluation.
Sources
- Heretic README (captured 2026-08-31)Source checked 2026-08-31
- Heretic repositorySource checked 2026-08-31