Qwen3-4B LiteRT-LM Models

This repository contains LiteRT-LM variants of Qwen/Qwen3-4B optimized for on-device text generation.

Available Artifacts

File	Quantization	Context	Size
qwen3_4b_channelwise_int8_float32kv.litertlm	channel-wise INT8 weights, float32 KV	4096	5.28 GB
qwen3_4b_mixed_int4.litertlm	TorchAO mixed INT4, float KV	2048	2535.88 MiB

Conversion Notes

The mixed INT4 .litertlm artifact was produced with a TorchAO-based quantize-first recipe from the original Hugging Face checkpoint. This is a mixed quantization layout rather than a uniform all-INT4 model: eligible linear projection weights are stored as blockwise INT4 with group size 32 and floating-point scales, token embedding weights use weight-only INT8 quantization, and normalization/reduction paths plus KV cache tensors remain floating point.

The mixed INT4 bundle also uses LiteRT-LM StableHLO composite ops for attention/cache execution, including odml.runtime_bmm and odml.cache_update.

qwen3_4b_mixed_int4.litertlm and qwen3_4b_channelwise_int8_float32kv.litertlm need litert-lm 0.18.0 or newer (tested on 0.18.0) and carry LiteRT-LM's official qwen3 configuration, so thinking is on unless enable_thinking is false, the reasoning arrives on the thought channel, and tool calls are parsed by the runtime. The context is 2,048 tokens in qwen3_4b_mixed_int4.litertlm, and with thinking on and no budget the thought can use it up before the answer: set thinking_token_budget (1024 was enough for a short arithmetic question) or enable_thinking=false. The context is 4,096 tokens in qwen3_4b_channelwise_int8_float32kv.litertlm.

Performance

Desktop benchmark: AMD Radeon AI PRO R9700, LiteRT-LM WebGPU, 256 prefill tokens, 32 decode tokens. Android rows use LiteRT-LM v0.13.1 with GPU OpenCL, 256 prefill tokens, and 64 decode tokens. Values report the warmed iteration from a two-iteration run unless noted.

Hardware benchmark disclosure: Results were measured by us on retail devices purchased through normal channels. These results are not affiliated with, sponsored by, endorsed by, or verified by Samsung, vivo, Qualcomm, MediaTek, Google, MLCommons, or Hugging Face. Results depend on device SKU, OS build, thermal state, battery mode, backend, model quantization, runtime version, and benchmark settings.

Device / Backend	Prefill (tok/s)	Decode (tok/s)	TTFT (s)	Peak Private Footprint
Desktop GPU WebGPU	1327.33	87.52	0.20	1697 MB
Samsung SM-S937U1 GPU OpenCL	357.57	19.14	0.77	1609 MB
vivo V2502A GPU OpenCL	170.66	12.67	1.58	4722 MB
TECNO LJ9 GPU OpenCL	104.37	10.86	2.54	4906 MB

Try It

Install uv and run:

uv tool install litert-lm
uvx litert-lm run --from-huggingface-repo=litert-community/Qwen3-4B qwen3_4b_mixed_int4.litertlm --prompt="What is the capital of France?"

Integration

Ready to integrate this into your product? Get started in the LiteRT-LM documentation.

Citation

@misc{qwen3technicalreport,
      title={Qwen3 Technical Report},
      author={Qwen Team},
      year={2025},
      eprint={2505.09388},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.09388},
}

Known runtime issues

From a conversation preface (the system message, initial messages, tools and extra context a conversation is created with) that has tools and no messages, LiteRT-LM renders these files' chat template with an empty message list, and the template reads messages[0] without checking that the list is non-empty, so the render fails with Failed to apply template: undefined value. This happens in two places: when the preface is prefilled at creation (prefillPrefaceOnInit = true in Kotlin, prefillPrefaceOnInit: true on the web, SetPrefillPrefaceOnInit(true) in C++; the default is false), where creating the conversation fails with that error, and in the preface render (renderPrefaceIntoString() in Kotlin and Swift, RenderPrefaceIntoString() in C++, litert_lm_conversation_render_preface_to_string in the C API). The same happens with extra context (such as enable_thinking) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2). Read in the source of LiteRT-LM 0.17.0, 0.18.1 and main; measured with LiteRT-LM 0.18.1 (the litert-lm package from PyPI) through its C API on these files' template text. The default path, where the first message is rendered together with the preface, is not affected: the Python API and the CLI of 0.18.1 expose neither the flag nor the preface render, and Google AI Edge Gallery sets neither (its source as read on 2026-10-25). A system message or any initial message in the conversation config avoids it.