Heretic
Heretic 性能与成本：基准完整实验而非单看 GPU 分钟
测量转换、Optuna 试验、量化、残差图、存储和人工复核。
你将学会
- Explain the KL/refusal optimization objective
- Run a pinned model experiment
- Design broader safety evaluation
开始前需要
- Basic Git and command-line usage
- Comfort reading a project README
You can explain the project, run its documented first step, and decide what to verify before adopting it.
先看结论
- The meaningful unit is an accepted, reproducible checkpoint.
- GPU transform and CPU research plots have different bottlenecks.
- Cost includes review, storage, and failed experiments.
Choose the right unit
以被接受的可复现 checkpoint 为单位，固定模型、提示、试验预算和硬件，分别记录 p50/p95、VRAM、CPU、字节、失败重跑和审核时间。
Freeze model revision, prompt manifests, trial budget, PyTorch/CUDA, GPU, batch policy, quantization, and output format. Report p50/p95 wall time, peak VRAM, CPU hours, storage, download bytes, trial count, and failure rate.
Separate pathways
Benchmark baseline inference, transformation, built-in evaluation, chat smoke tests, and research plots separately. The README's 20–30 minute RTX 3090 example is useful context for Qwen3-4B, while larger models or PaCMAP can change the cost profile dramatically.
Calculate cost per accepted checkpoint, including GPU time, CPU plotting, storage, review minutes, and re-runs after failed or unsafe outputs. Quantization may lower VRAM but should be measured for quality drift.
Optimization levers
Tune batch size only within memory and reproducibility limits; cap Optuna trials, cache immutable model weights, and schedule CPU-heavy plots asynchronously. Establish canaries and stop thresholds for capability regression or anomalous refusal changes.
如何选择
| 比较维度 | 方案 A | 方案 B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
实施步骤
- 1
Freeze workload, stack, hardware, and trial budget.
- 2
Measure cold/warm, transform, evaluation, and plot paths.
- 3
Record VRAM, CPU, bytes, failures, and review time.
- 4
Set canary and rollback thresholds before scaling.
可复制示例
manifest -> baseline p50/p95
transform + Optuna trials -> VRAM/time
eval + plots + review -> accepted checkpoint cost
canary drift -> stop / rollback常见问题
Is the README's 20–30 minute figure a guarantee?
No. It is a hardware- and model-specific example; reproduce it with matching conditions.
Should I maximize Optuna trials?
No. Set a budget and stop when quality, cost, or safety evidence no longer improves.
资料来源
- Heretic README (captured 2026-08-31)来源核查 2026-08-31
- Heretic repository来源核查 2026-08-31