Heretic
Heretic: rendimiento y coste de un experimento completo
Benchmark de transformación, trials, cuantización, plots, almacenamiento y revisión.
Qué aprenderás
- Explain the KL/refusal optimization objective
- Run a pinned model experiment
- Design broader safety evaluation
Antes de empezar
- Basic Git and command-line usage
- Comfort reading a project README
You can explain the project, run its documented first step, and decide what to verify before adopting it.
Conclusiones clave
- The meaningful unit is an accepted, reproducible checkpoint.
- GPU transform and CPU research plots have different bottlenecks.
- Cost includes review, storage, and failed experiments.
Choose the right unit
Mide por checkpoint aceptado: fija modelo, prompts, trials y hardware y registra p50/p95, VRAM, CPU, bytes, fallos y minutos de revisión.
Freeze model revision, prompt manifests, trial budget, PyTorch/CUDA, GPU, batch policy, quantization, and output format. Report p50/p95 wall time, peak VRAM, CPU hours, storage, download bytes, trial count, and failure rate.
Separate pathways
Benchmark baseline inference, transformation, built-in evaluation, chat smoke tests, and research plots separately. The README's 20–30 minute RTX 3090 example is useful context for Qwen3-4B, while larger models or PaCMAP can change the cost profile dramatically.
Calculate cost per accepted checkpoint, including GPU time, CPU plotting, storage, review minutes, and re-runs after failed or unsafe outputs. Quantization may lower VRAM but should be measured for quality drift.
Optimization levers
Tune batch size only within memory and reproducibility limits; cap Optuna trials, cache immutable model weights, and schedule CPU-heavy plots asynchronously. Establish canaries and stop thresholds for capability regression or anomalous refusal changes.
Cómo elegir
| Criterio | Opción A | Opción B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Pasos de implementación
- 1
Freeze workload, stack, hardware, and trial budget.
- 2
Measure cold/warm, transform, evaluation, and plot paths.
- 3
Record VRAM, CPU, bytes, failures, and review time.
- 4
Set canary and rollback thresholds before scaling.
Ejemplo para copiar
manifest -> baseline p50/p95
transform + Optuna trials -> VRAM/time
eval + plots + review -> accepted checkpoint cost
canary drift -> stop / rollbackPreguntas frecuentes
Is the README's 20–30 minute figure a guarantee?
No. It is a hardware- and model-specific example; reproduce it with matching conditions.
Should I maximize Optuna trials?
No. Set a budget and stop when quality, cost, or safety evidence no longer improves.
Fuentes
- Heretic README (captured 2026-08-31)Fuente verificada 2026-08-31
- Heretic repositoryFuente verificada 2026-08-31