mlx-community/Hy3-OptiQ-2bit

Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. All OptiQ quants · Docs

A 295-billion-parameter model that runs in 7.6 GB of RAM on a Mac. This is a 2-bit mixed-precision MLX quant of tencent/Hy3, produced by mlx-optiq. It is 88 GB on disk, and while the model generates only about 7.6 GB sits in RAM: attention, the router, the shared expert and the embeddings stay resident, and the 192 routed experts are read off the SSD as the router picks them.

What it is

Property	Value
Base	tencent/Hy3 (sparse MoE, 192 experts, 8 active per token, 80 layers)
Parameters	295 B
Bit-widths	2-bit routed experts; 6-bit attention; 8-bit shared expert, embeddings and LM head
Achieved bits-per-weight	2.39
On disk	88 GB
Resident while running	~7.6 GB (routed experts streamed)

No Capability Score is published for this quant. Running the six-benchmark suite against a model that decodes off SSD would take days, and at 2 bits on the routed experts the point of the artifact is different: that a 295 B MoE runs at all on consumer Apple Silicon.

Run it

Hy3 is not an architecture stock mlx-lm knows, so import optiq once to register it:

pip install "mlx-optiq>=0.4.15"

The routed experts are far too large to sit resident, so serve it with SSD expert streaming. optiq serve turns this on by itself for a MoE quant that would not fit in RAM (--stream-experts forces it):

optiq serve --model mlx-community/Hy3-OptiQ-2bit

That gives you an OpenAI and Anthropic compatible endpoint with mixed-precision KV cache, tool-call healing and prompt caching. Only the routed experts stream per token, so the footprint stays near 7.6 GB regardless of how large the model on disk is.

A fast SSD matters more than RAM here. Every token reads 8 experts per layer from disk, so decode speed tracks read throughput.

Notes

This is an extreme quant. Two bits on the routed experts is lossy, and anything where accuracy matters should use the bf16 weights or a higher-bit quant. What this one demonstrates is a model of this size running on a Mac, at a resident footprint that fits a 16 GB machine.

Links

• Project website: mlx-optiq.com
• All OptiQ quants: mlx-optiq.com/models
• PyPI: pypi.org/project/mlx-optiq
• Base model: tencent/Hy3