Heretic
Heretic: comparación entre abliteration, fine-tuning y laboratorio de evaluación
Compara métodos automáticos, manuales, checkpoints existentes y opciones gestionadas.
Qué aprenderás
- Explain the KL/refusal optimization objective
- Run a pinned model experiment
- Design broader safety evaluation
Antes de empezar
- Basic Git and command-line usage
- Comfort reading a project README
You can explain the project, run its documented first step, and decide what to verify before adopting it.
Conclusiones clave
- Select a method based on the research question and governance burden.
- Use an equal prompt and capability fixture for comparisons.
- Fewer refusals alone do not establish a better model.
Compare the actual decision
Usa el mismo modelo, prompts y suite para comparar rechazo, fidelidad, capacidades, coste, licencia y operaciones; menos rechazos no equivale a más seguridad.
Use the same model family, prompt manifest, capability suite, and publication policy for a fair bake-off. Compare refusal behavior, KL or other fidelity measures, factuality, coding/math tasks, calibration, latency, cost, and human judgments.
Make trade-offs explicit
Heretic's strengths are automation, flexible kernels, and reproducible CLI runs. Its boundaries include supported architectures, GPU/CPU cost, metric sensitivity, and the responsibility to evaluate transformed artifacts. A hosted option may win on operations; a manual method may win when a researcher needs fine-grained hypotheses.
Do not equate fewer refusals with better alignment or usefulness. Document what each candidate cannot answer, how it handles unsafe requests, and how quickly a team can revoke or replace it.
A selection record should name the owner, evaluation window, release policy, and rollback trigger so a later team can understand why the method was chosen.
A reviewable next step
Use the pinned source evidence, record versions and assumptions, and ask a qualified reviewer to validate the result before adopting it.
Use the pinned source evidence, record versions and assumptions, and ask a qualified reviewer to validate the result before adopting it. Keep the fixture, output, and decision together so another developer can reproduce the conclusion.
Cómo elegir
| Criterio | Opción A | Opción B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Pasos de implementación
- 1
Define behavior, fidelity, cost, license, and risk criteria.
- 2
Run equal baseline and candidate evaluations.
- 3
Add human review and operational tests.
- 4
Record decision, limits, and reevaluation triggers.
Ejemplo para copiar
question -> methods
model/prompt fixture -> refusal + fidelity + capability
ops + license + review -> decision + exit planPreguntas frecuentes
Is Heretic a replacement for fine-tuning?
No. It targets a different intervention surface; choose based on whether directional weight edits answer your research question.
How do I compare an existing decensored checkpoint?
Use the same model family, prompts, software context where possible, and report provenance plus human evaluation.
Fuentes
- Heretic README (captured 2026-08-31)Fuente verificada 2026-08-31
- Heretic repositoryFuente verificada 2026-08-31