Instructions to use Qwen/Qwen3-0.6B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Qwen/Qwen3-0.6B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Qwen/Qwen3-0.6B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
AMD Developer Cloud
Local Apps Settings

vLLM

How to use Qwen/Qwen3-0.6B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Qwen/Qwen3-0.6B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3-0.6B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Qwen/Qwen3-0.6B

SGLang

How to use Qwen/Qwen3-0.6B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Qwen/Qwen3-0.6B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3-0.6B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Qwen/Qwen3-0.6B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3-0.6B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Qwen/Qwen3-0.6B with Docker Model Runner:
```
docker model run hf.co/Qwen/Qwen3-0.6B
```

Very coherent for the size, though it is also adamant in defending its wrong conclusions and false knowledge

by cmp-nct - opened May 1, 2025

Discussion

cmp-nct

May 1, 2025

•

edited May 1, 2025

I found the model to be pretty smart for being so small in size, it's remarkable coherent and it is multi-lingual - similar maybe to A1 in language knowledge.

I doubt that reasoning training for such small models is very helpful, I think they are better in non reasoning tasks.
Reasoning certainy will make them smarter but they stay so dumb that it makes no difference in real world use.

The major flaw I found was that the model is insisting on wrong knowledge.
When training a tiny llm I believe you should change the approach, the model needs to be aware that it is very "uncertain" bout literally everything.

For example, Qwen 0.6B (and also some larger versions) claims that Einstein died in Germany, never was in the USA and it actually got the date correct.
However if you then ask if how einstein "survived" the WW2 in Germany, Qwen says that he died before WW2 heated up.
Asking it for dates it will show correct dates for WW2 and for Einstein but it will always defend the conclusion that Einstein died before.

While this might be solveable with reasoning, the underlying flaw is that certainty it is being trained for during fine tuning.
I think it would be smart to try add higher levels of uncertainty into tiny models during post training.
Their natural response should not be certain which might also make it less prone to defend wrong knowledge or bad logic.

oamazonasgabriel

Jun 20

@cmp-nct - I've been experimenting it with tool calling and so far it's pretty decent indeed.

As per your uncertainty comment - I thing an Agent system prompt tailored for that bias should be a step in that direction - would need proper evals though.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment