MiMo-V2.6-Flash-REAP50

This is XiaomiMiMo/MiMo-V2.6-Flash-RL with half of its routed experts removed: each of the 47 MoE layers goes from 256 experts to 128. The weights come to 86.1 GiB in 65 safetensors shards, small enough to fit a single 128 GB unified-memory machine (NVIDIA Jetson AGX Thor). That fit is an estimate, not a test. It is meant for people who want the omni-modal MiMo-V2.6-Flash model on one box of that size.

Status: never served, never benchmarked. The checkpoint was published without being run for generation. There is no serve log, no quality evaluation, no needle-in-a-haystack test and no measured throughput on any hardware. Any decode speed you may have seen attached to this release (for example "~5 tok/s on Thor") came from memory-bandwidth arithmetic. Nobody measured it.

Model overview

Base model	XiaomiMiMo/MiMo-V2.6-Flash-RL (MIT)
Architecture	MiMoV2ForCausalLM (model_type: mimo_v2), custom code included (modeling_mimo_v2.py)
Parameters	about 157B in the language model after pruning (about 309B before); about 15B active per token. These are computed from the safetensors headers, not taken from a card (see the note below)
Layers	48: layer 0 is dense, layers 1–47 are MoE. 9 full-attention layers (0, 5, 11, 17, 23, 29, 35, 41, 47) and 39 sliding-window layers (window 128)
Experts	128 of 256 routed experts kept per MoE layer, top-8 routing, no shared expert (sigmoid scoring, noaux_tc, n_group 1)
Attention	64 query heads; 4 KV heads on full-attention layers and 8 on SWA layers; head dim 192 (K) / 128 (V)
Context length	1,048,576 in config.json. Not tested: calibration sequences were at most 4,095 tokens
Vocabulary	152,576, untied embeddings
MTP	3 next-token-prediction layers (num_nextn_predict_layers: 3) in model_mtp.safetensors. They have dense FFNs, so pruning did not touch them
Input modalities	text, image, video and audio (vision tower, audio encoder, and audio_tokenizer/). The towers are unchanged. None of these modalities has been tested on the pruned model
Size on disk	92,448,660,520 bytes = 86.10 GiB of model weights (64 model_pp0_ep*_shard0.safetensors + model_mtp.safetensors). The whole repository is 87.86 GiB once you add audio_tokenizer/model.safetensors (1.74 GiB) and the tokenizer files
Licence	MIT, inherited from the base model

Parameter note: 6,016 expert tensors (47 layers × 128 experts) × 3 projections × 4096 × 2048 = 151.4B expert parameters, plus 6.0B non-expert language-model parameters (attention, layer-0 FFN, routers, embeddings, LM head). Restoring the 128 dropped experts per layer gives 308.8B, which matches the base model's stated 309B. Active parameters (8 experts per layer plus all non-expert weights) are about 15.4B, the same as the base, because pruning does not change top-k. The vision tower (0.73B), audio encoder (0.26B) and MTP layers (0.99B) are counted separately.

Precision per component (from the safetensors headers)

Component	Stored as	Changed by this release?
Routed experts (mlp.experts.*)	MXFP4: packed E2M1 nibbles (U8) + E8M0 block scales (U8), block 32. This is Xiaomi's source format	Dropped or kept whole. Kept experts' bytes are copied unchanged and renumbered 0–127
Routers (mlp.gate.weight, e_score_correction_bias)	BF16 / F32	Sliced to the 128 kept rows. Values unchanged (see Routers)
Attention qkv_proj	FP8 E4M3 with F32 128×128 block scales	No
Attention o_proj	BF16 (excluded from FP8 at source)	No
Layer-0 dense FFN	FP8 E4M3 with block scales	No
Embeddings, LM head, norms	BF16	No
Vision tower, audio encoder	BF16	No
MTP layers	FP8 E4M3 (qkv_proj, FFN) + BF16	No

Experts already ship as 4-bit MXFP4 in the source checkpoint, so re-quantizing them buys almost nothing. Pruning is the only real compression lever, and this release uses only that.

Which version should I use

Repository	Format	Size	Notes
MiMo-V2.6-Flash-REAP50 (this repo)	safetensors: MXFP4 experts, FP8 / BF16 elsewhere	86.10 GiB weights	The only artifact here that keeps the model's native video path. Needs a runtime with MiMo-V2 support and a 128-expert config. Untested
MiMo-V2.6-Flash-REAP50-GGUF	GGUF MXFP4_MOE (86.06 GiB) and a mixed MXFP4/Q2_K file named Q2_K (61.64 GiB), plus mmproj and mtp GGUFs	61.6–86.1 GiB	For llama.cpp. Text, image and audio at most: llama.cpp has no native path for MiMo video. Never loaded

Nothing has been measured, so neither repo can be recommended on quality or speed.

Quickstart

No inference command has been run against this checkpoint. All that is known to work is downloading it:

hf download patrickbdevaney/MiMo-V2.6-Flash-REAP50 --local-dir MiMo-V2.6-Flash-REAP50

The repo ships Xiaomi's remote code (modeling_mimo_v2.py, configuration_mimo_v2.py, auto_map in config.json) and the base model's chat template and processor config. A runtime that loads the base MiMo-V2.6-Flash-RL checkpoint should load this one, because only the expert count changed (n_routed_experts: 128). That is untested. It needs about 86 GiB for the weights alone, before any KV cache.

Evaluation

Not yet evaluated. This checkpoint has no end-to-end measurement of any kind:

• no perplexity, no teacher-forced agreement against the unpruned model, and no generative or task benchmark;
• no image, audio or video test;
• no long-context or needle-in-a-haystack test;
• no throughput or memory measurement.

The only numbers that exist are selection-time diagnostics on the calibration corpus (see How it was made). They measure how much routed output mass the kept experts carry on the data used to pick them. They are not a quality result and should not be read as one.

Hardware & memory

• Produced on: one NVIDIA Jetson AGX Thor (128 GB unified memory). That machine ran the calibration pass, the expert selection and the checkpoint write.
• Run on: nothing. The model has not been loaded for inference anywhere.
• Fit estimate (arithmetic only). The 86.1 GiB of weights plus an FP8 KV cache at the full 1,048,576 tokens comes to about 97.4 GiB before runtime workspace. The KV figure is about 11.25 GiB: 9 full-attention layers × 4 KV heads × (192 + 128) dims × 1 byte, about 11.5 KB per token, while the 39 SWA layers hold only a 128-token window. That total sits under the roughly 117 GiB usable on Thor, but it has not been tried. A BF16 KV cache would double the KV term.

How it was made

1. Calibration. Expert statistics were accumulated over 49.0M tokens in 27 chunks of packed 4,096-token rows. Every document was truncated at 4,095 tokens. The pass ran layer by layer through the unpruned model, one layer resident at a time. The corpus has nine domains:

   domain	tokens		domain	tokens
   agentic	11,501,033		image	3,001,238
   ballast (general)	10,003,073		science	2,980,200
   code	9,503,748		audio	1,502,892
   math	7,502,972		video	1,502,610
   finance	1,503,455

   The image, audio and video domains are real media run through the model's own towers and processor (with their captions), 6.0M tokens in all, rather than text captions alone. Sources were limited to permissively licensed datasets, with licences checked against the Hugging Face API; a source with no declared licence was excluded. Non-English coverage was not measured separately.

2. Selection. Per-expert saliency used the REAP criterion (reap_1_1_1, router-weighted expert activation; Lasby et al., 2025). The prune set was solved with the HOPE objective (arXiv 2609.18916). HOPE minimises pᵀFp, the output error of a prune set, interaction terms between experts included; REAP corresponds to the same objective with those off-diagonal terms set to zero. The ratio was 0.50, uniform across all 47 MoE layers (128 kept per layer). The top 8% of experts in each domain were protected from pruning (protect_frac 0.08), and candidates were compared by their worst domain rather than their mean.

   Selection-time diagnostics for the published mask (artifacts/masks/mask.json in the pipeline repo). These are retained routed output mass on the calibration corpus, a proxy and not an evaluation:

   domain	retained		domain	retained
   audio (worst)	0.9935		finance	0.9974
   image	0.9937		ballast	0.9976
   video	0.9938		code	0.9983
   science	0.9973		agentic	0.9983
   math	0.9974		mean	0.9964

   HOPE objective pᵀFp = 0.01624.

3. Routers. A router-only refit by output matching against the unpruned teacher was tried on all 47 layers. In every layer the refit failed to beat simply slicing the teacher's router (improvement +0.0%), so all 47 routers are the teacher's weights sliced to the kept experts. Only about 1.1% of routed slots in the calibration data had gone to experts that were pruned.

4. Write-out. Whole expert tensors were dropped and the survivors renumbered, with each router's row order checked against the new numbering. No weights were re-quantized and no healing or fine-tuning was applied.

Code, gates and the method write-ups are in the pipeline repository: patrickbdevaney/xiaomi-2.6-flash-REAP.

Limitations

• Unevaluated. Nothing shows that this model performs as well as the original. Pruning 50% of experts is lossy and cannot be undone, and losses can be concentrated in one capability even when the averages look fine.
• Long context is unverified. Calibration never saw more than 4,095 tokens, and the 1M figure is config plus memory arithmetic.
• Multimodal inputs are untested. The towers are unchanged, and the calibration fed real image, audio and video through them. No multimodal output from the pruned model has been checked.
• Draft heads. The base repo's separate dflash/ speculative-decoding drafter is not included: it was trained against the unpruned model, and nobody has checked it against this one. The 3 native MTP layers are included unchanged; their acceptance rate on the pruned trunk has not been measured.
• Calibration covered the nine domains above. Other domains and languages were not measured.

Changelog / corrections

• 2026-09-25: checkpoint published (weights unchanged since then).
• 2026-10-24: correction: never served or benchmarked; "~5 tok/s" was bandwidth arithmetic; the 1M-context fit is sizing arithmetic; per-domain retention is a selection-time proxy. No weights were changed.
• 2026-11-06: card rewritten and verified against repo files and build artifacts; a 2026-10-26 automated rewrite contained errors.

License & attribution

MIT, inherited from the base model. The base model, its architecture, its multimodal towers and its MTP layers are the work of the Xiaomi MiMo team: XiaomiMiMo/MiMo-V2.6-Flash-RL. This is an independent community derivative, not affiliated with or endorsed by Xiaomi. The expert-saliency criterion is REAP (arXiv 2510.13999); the prune-set objective is HOPE (arXiv 2609.18916).

Citation

@misc{devaney2026mimov26flashreap50,
  title        = {MiMo-V2.6-Flash-REAP50: 50\% routed-expert pruning of MiMo-V2.6-Flash-RL},
  author       = {Patrick Devaney},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/patrickbdevaney/MiMo-V2.6-Flash-REAP50}}
}