Awesome Modell - Thanks a lot! MTP/DFlash/DSpark planned?

#2
by GxBender - opened

I really love it.
I've built a local AI Assistant on a GTX 1060 and an i5 (6x 2.8GHz) — I was really stuck trying to find suitable models that could handle my code for my side projects.

Qwen2.5-9B failed on my code, but your merge nailed it. Exactly what I was searching for. I'm going to use it for complex coding as well as agent tasks / tool use.

Can you tell me if you have MTP, DFlash, or DSpark support planned for this model?

Keep up the great work, really appreciate it

Thanks for the positive feedback!
Speculative decoding and MTP shine mostly on larger models (27B–70B+) where generation is heavily memory-bandwidth bound. On smaller models, including this merge, generation is already fast out of the box. Adding MTP layers introduces an extra ~1–2 GB VRAM overhead and verification compute that yields diminishing returns. That extra VRAM is usually better spent on higher quantizations or longer context windows.

Ah, I was actually using this one:
https://huggingface.co/pragmaticcs/Triumvirate-Qwopus-MiMo-Ornith-9B-Coder-GGUF
(I had the wrong tab open, it was late yesterday 😂)

Are you planning to release more lower quants (or standard K-quants like Q4_K_M/ Q4_K_S/ Q3_K_L) or lower Imatrix options (IQ4_XS, IQ3_M, IQ3_S, IQ3_XS)?
I'm currently using your native IQ4_NL GGUF and the reasoning/coding quality is absolutely top-tier! However, on my Pascal GPU (GTX 1060), I'm hitting ~13 T/s (with speculative decoding). I'd love to test slightly lower or standard K-quants to see if I can push the speed even further if the quality drop isn't too big.
Thanks again for the great work on this merge! 😊

Edit: Especially with very high context the model is still working great, not getting halluzinations and still coding perfect. I used it to split very big python scripts into modularized versions and the conversations had 25-50 K Context - it still worked with ~9-11 T/S. ](Edit: Especially with very high context, the model is still working great — no hallucinations and still coding perfectly. I used it to split very big Python scripts into modularized files with conversations around 25–50k context, and it still maintained ~9–11 T/s!)

On my benchmarks Q5_K_M, IQ4_NL and I1.Q5_K_M were the ones that performed the best, which is why I didn't create the full range of quants. I try to test out most of the quants on my machine to minimize the chances of releasing a broken model. However, you can create suitable static and imatrix quants based on your system specifications using this space for free: https://huggingface.co/spaces/ggml-org/gguf-my-repo

Sign up or log in to comment