Heretic Architecture: Residual Directions, Weight Kernels, and Optuna

A layered mental model

The README exposes four useful layers: prompt groups produce first-token residuals; layer-wise differences define directions; supported attention out-projection and MLP down-projection matrices are orthogonalized; Optuna searches parameters against refusals and KL divergence.

The special `per layer` direction index selects each layer's direction, while a float index interpolates between neighboring directions. This turns a discrete set of measured vectors into a larger search space, but also makes experiment provenance essential.

The ablation kernel

`max_weight`, `max_weight_position`, `min_weight`, and `min_weight_distance` describe a flexible weight shape over layers. Attention and MLP components receive separate parameters because the README reports that MLP interventions can damage capabilities more.

The optimization objective is a co-minimization: fewer refusals on harmful prompts and low KL divergence on harmless prompts. Neither term is a complete model-quality objective; add capability, calibration, bias, and misuse evaluations around it.

Research branch

Optional research features compute residual geometry, use PaCMAP to project layers, align consecutive projections, and emit PNG/GIF artifacts. Keep this CPU-heavy analysis off the critical transform path and preserve the source arrays or hashes used to create plots.