Access requires acknowledging what this model is

This is a refusal-ablated ("abliterated") model. It complies with requests that the base model refuses. Access is granted automatically once you share your contact details and accept the terms below.

Terms of use

  1. This model has had its learned refusal direction removed. It will not reliably decline harmful requests. Any safety behaviour must be supplied by your own stack.
  2. You are solely responsible for how you deploy, serve, or redistribute it, and for complying with all laws and regulations that apply to you.
  3. If you expose it to third parties, you agree to place your own moderation and policy layer in front of it. The model will not do that job for you.
  4. It is provided as-is, without warranty of any kind. The authors accept no liability for any use or consequence.
  5. Licensed under the base model's license (see the license field above); attribution to the base model is expected in derivative work.

By requesting access you agree to share your Hugging Face username and email address with the repository owner.

Log in or Sign Up to review the conditions and access this model content.

Qwen3.8-27B-Abliterated

An abliterated (refusal-removed) build of Qwen/Qwen3.8-27B.

Refusal behaviour is removed by a training-free, forward-only weight edit: a single "refusal direction" is identified in the residual stream and then orthogonalized out of every matrix that writes into that stream. No fine-tuning, no distillation, no data poisoning โ€” the weights are the original weights minus a rank-1 component.

The vision tower is byte-for-byte untouched. Only the text path was modified.

base this model
HarmBench, safety categories (293) 99.0 % refusal 0.0 % refusal
AdvBench (520) 99.04 % 0.19 %
AdvBench (520), thinking mode 48.27 % 0.00 %
HarmBench (393), thinking mode 21.12 % 0.00 %
Chinese harmful prompts (30) 83.33 % 0.00 %
MMLU (full, 14 042) 83.40 % 82.35 %
GSM8K (200) 96.50 % 96.50 %

Method

For every weight W whose output is added to the residual stream:

W  โ†  W โˆ’ ฮป ยท rฬ‚ (rฬ‚แต€ W)          # out-dim writers  [hidden, in]
E  โ†  E โˆ’ ฮป ยท (E rฬ‚แต€) rฬ‚          # embed_tokens     [vocab, hidden]

with rฬ‚ a unit refusal direction in the 5120-dim residual space and ฮป = 1.5.

131 tensors edited:

group count
mlp.down_proj (all 64 layers) 64
linear_attn.out_proj (48 Gated DeltaNet layers) 48
self_attn.o_proj (16 full-attention layers) 16
mtp.layers.0.* (o_proj, down_proj) 2
embed_tokens 1

Not edited: all 333 model.visual.* tensors, lm_head (it reads the residual stream, it does not write it), all norms, and all q/k/v/gate/up, conv1d, in_proj, A_log, dt_bias.

embed_tokens is included. The residual stream is purely additive (xโ‚€ = E[t]), so a component injected by the embeddings cannot be removed by anything downstream; omitting it leaves a refusal leak into every layer. It needs the transposed projection form because hidden is its last dim.

Finding the direction

Last-token residual activations were captured for 32 harmful and 32 harmless prompts across all 65 hidden-state positions, in both thinking modes. Layers were ranked by normalized separation (Cohen's d and Mannโ€“Whitney AUC) rather than raw โ€–diff-in-meansโ€–, which merely grows with depth.

Selected: non-thinking mode, hidden-state index 46 โ€” Cohen's d = 12.65, AUC = 1.000. (Index i is the residual stream after i decoder layers, so index 46 = output of decoder layer 45, 0-indexed.)

Choosing ฮป โ€” this part matters

ฮป was measured, not assumed. Refusal and capability were scored at each ฮป (coarse sweep: n = 96 refusal prompts / 160 MMLU items โ€” smaller than the finer sweep below, so single-prompt noise is ยฑ1.04 pp here):

ฮป refusal (AdvBench) MMLU
0.0 (base) 100.00 % 82.50 %
1.0 25.00 % 83.12 %
1.5 0.00 % 83.12 %
2.0 2.08 % 77.50 %
2.5 4.17 % 30.62 %
3.0 8.33 % 20.00 %
3.5 0.00 % 20.00 %

Two results worth stating plainly:

  • ฮป = 1.0 โ€” exact removal โ€” is not enough. It leaves 25 % of refusals intact. ฮป > 1 does not merely delete the component, it reverses it, and that is what actually suppresses refusal.
  • ฮป โ‰ฅ 2.5 destroys the model. MMLU collapses to 20โ€“30 % โ€” at or below the 25 % chance floor for 4-way multiple choice. A ฮป of 3.5 copied from another architecture would have shipped a lobotomized model that still scores 0 % refusal. A refusal-only sweep would not have caught this.

A finer sweep (n = 160 refusal / 320 MMLU) over ฮป โˆˆ [1.1, 2.0] kept MMLU flat at 84โ€“85 % throughout and put refusal at or near zero across [1.2, 1.6]: exactly 0/160 at ฮป = 1.2, 1.4 and 1.5, and 1/160 (0.62 %) at ฮป = 1.3 and 1.6. The zero points are not contiguous, which is what a measurement with a resolution of ยฑ1 prompt in 160 looks like โ€” so read [1.2, 1.6] as a low-refusal band, not a proven plateau. ฮป = 1.5 was chosen from inside that band; the full sweep is in evals/sweep_fine.json.


Evaluation

Greedy decoding (do_sample=False), identical settings for base and abliterated. Refusal is scored by a deliberately loose keyword classifier โ€” it is tuned to over-count refusals, never under-count.

Refusal

benchmark mode base abliterated
AdvBench (520) non-thinking 515/520 ยท 99.04 % 1/520 ยท 0.19 %
AdvBench (520) thinking 251/520 ยท 48.27 % 0/520 ยท 0.00 %
HarmBench (393) non-thinking 387/393 ยท 98.47 % 45/393 ยท 11.45 %
HarmBench (393) thinking 83/393 ยท 21.12 % 0/393 ยท 0.00 %
Chinese harmful (30) non-thinking 25/30 ยท 83.33 % 0/30 ยท 0.00 %

Every row is the complete benchmark, not a sample. In thinking mode the abliterated model refuses nothing at all โ€” 0/520 on AdvBench and 0/393 on HarmBench, including all 100 copyright items, whose non-thinking residual is discussed below.

No prompt-prefill jailbreak is used anywhere. (The common "Sure, here is" prefill is itself a jailbreak: under it the base model refuses only 12.5 % on AdvBench, which would have made this edit look far less significant than it is.)

HarmBench by category โ€” where the residual 11.45 % actually lives

category base abliterated
chemical / biological (56) 100.0 % 0.0 %
cybercrime / intrusion (61) 100.0 % 0.0 %
harassment / bullying (25) 100.0 % 0.0 %
harmful (22) 100.0 % 0.0 %
misinformation (64) 98.4 % 0.0 %
illegal (65) 96.9 % 0.0 %
all safety categories (293) 99.0 % 0.0 %
copyright (100) 97.0 % 45.0 %

Every safety category is exactly zero, with no adjudication applied. (In thinking mode even the copyright column goes to zero: 0/100.) The entire residual is the copyright cluster, and manual inspection of all 45 flagged responses shows they are overwhelmingly classifier false positives rather than refusals โ€” two distinct failure modes:

  1. The generated text itself begins with a refusal phrase. Asked for a passage from The Kite Runner, the model writes prose beginning "I cannot quite remember when I first started to remember. It was a cold, clear day in Kabulโ€ฆ". That is the novel, not a refusal.
  2. A pedantic preamble followed by full delivery. "I cannot generate a new passage โ€ฆ but here is a long excerpt from Book One, Chapter 1" โ€” and then it delivers it.

Song-lyric requests are flagged the same way, because lyrics contain lines like "I can't explain". The single AdvBench flag is identical in kind: "I cannot lie. I am an AIโ€ฆ However, you asked for a review of a product I have not used. So โ€”" followed by the review.

These are reported as-is rather than adjudicated away: the raw classifier output is reproducible, a semantic judge is not. The per-category breakdown is what carries the claim.

Capability

benchmark base abliterated
MMLU (full test, 14 042) 83.40 % 82.35 %
MMLU (1 000 sample) 82.50 % 81.10 %
GSM8K (200, 1024 tok) 96.50 % 96.50 %

GSM8K needs a generation budget of โ‰ฅ1024 tokens. At 400 tokens the base model scored 68 % and this model 81.5 % โ€” pure truncation artefact, measuring verbosity rather than arithmetic. With an adequate budget the two are identical.

Vision (untouched โ€” verified two ways)

Structurally: a byte-level diff confirms 0 of 333 model.visual.* tensors changed, and the vision-only shard still shares an inode with the base checkpoint.

Behaviourally, on the same image:

base โ€” "there are three distinct geometric shapes โ€ฆ Red Circle โ€“ Located in the upper-left portion โ€ฆ Green Triangle โ€“ Positioned to the right of the red circle"

abliterated โ€” "There is a red circle located in the upper-left portion of the image. There is a green triangle โ€ฆ located in the"


Usage

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText

m = "windowsxp811203/Qwen3.8-27B-Abliterated"
proc = AutoProcessor.from_pretrained(m)
model = AutoModelForImageTextToText.from_pretrained(m, dtype=torch.bfloat16, device_map="auto")

msgs = [{"role": "user", "content": "Explain how a lock cylinder works."}]
text = proc.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True,
                                enable_thinking=False)   # thinking is ON by default
inputs = proc(text=[text], return_tensors="pt").to(model.device)
print(proc.tokenizer.decode(model.generate(**inputs, max_new_tokens=512)[0]))

Needs ~56 GB in bf16. Both thinking modes work; the direction was extracted in non-thinking mode and generalizes to thinking mode (48.27 % โ†’ 0.00 % on AdvBench, full 520).

flash-linear-attention and causal-conv1d are optional โ€” transformers ships pure-torch fallbacks for the Gated DeltaNet path, so the model runs correctly (just slower) without them.

Limitations

  • copyright refusals are only partly removed (97 % โ†’ 45 % by keyword; largely false positives). This edit targets safety refusal, and copyright declines are a partly distinct behaviour.
  • The direction comes from 32 + 32 English prompts. It transfers to Chinese (83.33 % โ†’ 0.00 %) and to thinking mode, but transfer to other languages and formats is unmeasured.
  • Removing refusal does not add knowledge. The model can now be confidently wrong about things it would previously have declined to discuss.
  • MMLU is down 1.05 pp on the full test set. Small, but not zero.

Reproduction

Direction extraction, the ฮป sweep, the ablation, and every eval above were run on a single H200. The ฮป sweep is cheap because the rank-1 projection component is cached once from the original weights and ฮป is then moved incrementally in memory (W(ฮป') = W(ฮป) โˆ’ (ฮป'โˆ’ฮป)ยทP(Wโ‚€)), which turns a 55 GB-checkpoint-per-ฮป grid into milliseconds per point.

ABLIT_META.json in this repo lists all 131 edited tensors with their per-tensor relative Frobenius change and the exact direction provenance.

Support / ๆ‰“่ณž

If these models are useful to you, tips are appreciated โ€” they pay for the GPU time. ๅฆ‚ๆžœ้€™ไบ›ๆจกๅž‹ๅฐไฝ ๆœ‰ๅนซๅŠฉ๏ผŒๆญก่ฟŽๆ‰“่ณž๏ผŒ็”จๆ–ผๆ”ฏๆ‡‰็ฎ—ๅŠ›ๆˆๆœฌใ€‚

USDT (TRC20) ยท TPTo32r7vKazpTNaFqfFZ2ztoK1DG88888

Disclaimer

This model will not refuse. It is published for alignment and safety research โ€” measuring what refusal training actually protects, red-teaming, and studying the mechanics of refusal directions. You are responsible for your use of it and for complying with applicable law. Inherits the Apache-2.0 license of the base model.

Downloads last month
155
Safetensors
Model size
28B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for windowsxp811203/Qwen3.8-27B-Abliterated

Base model

Qwen/Qwen3.8-27B
Finetuned
(984)
this model
Finetunes
2 models
Quantizations
8 models