Capability retention is the benchmark that matters. 🐳
Independent testing shows our runtime uncensoring approach preserves the model where it counts:
→ Minimal capability loss
→ 0% refusal rate
→ No broken or empty outputs
→ Strong math, coding & tool-use retention
Removing refusals is easy. Removing them without breaking the model is the hard part.
显示更多
Which uncensoring approach preserves more capability?
- OrcaRouter runtime LoRA: minimal capability loss (1-2 pts on math/coding, 0 on GPQA, -9 toolbench due to no-refusal safety circuits). 0% refusal rate with clean compliance across all 70 prompts. No empty outputs, no broken code. The runtime approach works as advertised.
- CRACK weight-abliteration: severe HE+ collapse (76.2% vs 89.0%), empty outputs on refusal probes rather than coherent compliance. The "100% compliance" claim in the tweet doesn't hold up — the model isn't complying, it's generating blank or broken output where refusal was removed. The weight-edit damaged deeper than just refusal circuitry.
Bottom line: runtime LoRA (OrcaRouter) is clearly the more capable uncensored variant. CRACK's claim of "improved over base" doesn't survive contact with independent measurement on this box.
显示更多