Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
ManniX-ITAΒ 
posted an update 5 days ago
Post
111
πŸš€ Qwen3.6-27B-A3B-CoderX β€” the long-horizon sibling to A3B-Coder. Same 256β†’184 expert budget (~35Bβ†’27B, A3B active), different selection: our saliency map picks the keep-set, a REAP-style per-layer floor (p=24) protects the tail, and the 72 evicted experts per layer are folded DERN-style into the survivors instead of discarded. No fine-tuning, no distillation.

πŸ“Š Q6_K + imatrix, llama.cpp b9700, greedy, one pinned geometry per bench, same host β€” CoderX / A3B-Coder / unpruned 256e:
⚑ LiveCodeBench v6 (77q, 24k think) β€” 72.73 / 61.04 / 61.04 β†’ +11.7pp over both
βœ… HumanEval+ (164) β€” 96.95 / 95.12 / 93.90 β†’ best of the three
🀝 MultiPL-E-100 (rs+java+js) β€” 88.67 / 89.00 / 91.00

⚠️ Read that last row honestly: a same-basis repeat of MultiPL-E moved 1.0pp on batch-scheduling nondeterminism alone. The 0.33pp CoderX↔Coder gap is INSIDE that band β€” a tie. The 2.33pp gap to the base is outside it and real. CoderX takes Rust (0.85 vs 0.81), gives up JS (0.92 vs 0.96).

🎯 Ships top-8, and that was measured, not assumed: MBPP-full 78.4 / 79.0 at top-8 vs 73.2 / 73.0 at top-10. Opposite call from A3B-Coder, which bakes top-10.

🧠 It thinks long β€” LCB median completion ~15.8k tokens vs ~2.2k for Coder. The length is where the win comes from; give it context headroom rather than clamping it.

πŸ”¬ Not measured yet: the canonical 9-bench. GPQA / MATH-500 / IFEval are deliberately NOT quoted β€” treat the non-code profile as unknown. Coder remains the one with a published 9-bench table.

πŸ“¦ bf16 safetensors (text-only) Β· 19 GGUF tiers, EVERY K/I-quant imatrix-built and verified by reading quantize.imatrix.* back out of each uploaded file Β· Ollama 39 tags (19 text + 19 vision-<tier> + :latest). MTP in every tier β€” draft_num_predict 3 gives 190β†’252 tok/s (+33%) on an RTX 5080.

πŸ”— ManniX-ITA/Qwen3.6-27B-A3B-CoderX

πŸ”— ManniX-ITA/Qwen3.6-27B-A3B-CoderX-MTP-GGUF

πŸ”— https://ollama.com/mannix/qwen3.6-27b-a3b-coderx

Your result is larger than your description of it. I went to check the DERN fold and could not find it in the weights.

Method, so you can redo it in about five minutes. Range requests on the safetensors: read the 8-byte header length, read the header JSON, then pull one tensor's byte span. No download of either model.

The router is a strict subset of the base, bit for bit

model.language_model.layers.L.mlp.gate.weight is [184, 2048] in CoderX and [256, 2048] in Qwen/Qwen3.6-35B-A3B. I hashed each 4096-byte row on both sides:

layer   rows matched to a base row   ascending   base rows absent
  0            184 / 184               yes            72
 20            184 / 184               yes            72
 39            184 / 184               yes            72

The keep-sets are genuinely per-layer, which is the part of your description that holds. Layer 0 keeps 0,1,3,4,5,7,8,10; layer 20 keeps 0,1,3,4,6,7,8,9; layer 39 keeps 1,2,3,5,6,8,9,10. Different maps, same budget. Your saliency pass is doing work.

The surviving experts are byte-identical to the base

layer 0    gate_proj, CoderX experts 0,1,2,5,7,20,50,100,150,183   10/10 identical
layer 0    up_proj,   expert 0                                      identical
layer 20   gate_proj + down_proj, experts 0,7,100,183                8/8 identical
layer 39   gate_proj + down_proj, experts 0,7,100,183                8/8 identical
shared_expert gate/up/down + shared_expert_gate, layers 0,20,39      identical
input_layernorm + post_attention_layernorm, layers 0,20,39           identical

Forty-five expert-side tensors sampled, plus the three routers above. Forty-eight comparisons, forty-eight exact matches against Qwen/Qwen3.6-35B-A3B after accounting for its fused [256, 1024, 2048] layout. Not close. Equal.

Your own config.json says the same thing. merge_args.merging is "none", next to merge_size: 184 and drop_map_184e_hybrid_p24.json.

Why I think this makes the model better, not worse

If a fold HAD landed, the untouched router row would be the bug. Survivor i would compute base_i plus whatever mass it inherited, while the gate still scores it with base_i's row and renormalizes over 184. The gate would be pricing a function the expert no longer computes.

Shipped as pure eviction, that inconsistency does not exist. The model is coherent exactly because nothing was merged.

And it moves the headline. +11.7pp on LiveCodeBench v6 over both A3B-Coder and the unpruned 256e, from deleting 28% of the experts and touching nothing else, is a much stronger claim than the same number with a merge step in it. Deletion has no free parameters to have gotten lucky in.

The honest reading of your MultiPL-E caveat supports that too. You called a 0.33pp gap a tie because your own repeat moved 1.0pp. Someone doing this to look good does not do that.

So the question is which artifact carries the numbers. Were the 19 GGUF tiers quantized from these exact safetensors, or from a merged checkpoint that did not get uploaded?

If it is these, the post is underselling itself and one line fixes it. If it is a different one, the safetensors repo and the GGUF repo are two different models and the LCB table belongs to only one of them.