doesn't work in vllm
ERROR 08-08 19:42:07 [core.py:632] File "/home/ubuntu/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 249, in _load_module
ERROR 08-08 19:42:07 [core.py:632] yield from self._load_module(prefix,
ERROR 08-08 19:42:07 [core.py:632] File "/home/ubuntu/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 222, in _load_module
ERROR 08-08 19:42:07 [core.py:632] loaded_params = module_load_weights(weights)
ERROR 08-08 19:42:07 [core.py:632] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-08 19:42:07 [core.py:632] File "/home/ubuntu/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_moe.py", line 477, in load_weights
ERROR 08-08 19:42:07 [core.py:632] param = params_dict[name]
ERROR 08-08 19:42:07 [core.py:632] ~~~~~~~~~~~^^^^^^
ERROR 08-08 19:42:07 [core.py:632] KeyError: 'layers.4.mlp.gate.g_idx'
Process EngineCore_0:
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/envs/vllm/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/ubuntu/miniconda3/envs/vllm/lib/python3.12/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/ubuntu/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 636, in run_engine_core
raise e
Which vllm version? If you build from source I think it should work. These MoE models required some changes that were incorporated recently.