Q-50M-Base
A compact gated decoder language model pretrained from scratch
Q-50M-Base is a base language model, not a chat or instruction model. It is designed for text continuation and as a starting point for fine-tuning. It should not be expected to reliably follow user instructions.
Model summary
Q-50M-Base is a 50.9M-parameter decoder-only causal language model developed by Q-Project. It was pretrained from scratch on approximately 5 billion tokens from FineWeb-Edu and does not inherit weights from another model.
The model keeps the well-tested shape of a modern Transformer decoder while adding three inexpensive mechanisms intended to improve stability and parameter efficiency at small scale:
- Grouped-query attention (GQA) with 8 query heads and 2 key/value heads.
- Per-head QK-Norm before the attention score calculation.
- Content-dependent scalar gates on both the attention and MLP residual branches.
The implementation is built on the Hugging Face Mistral classes for compatibility with the Transformers generation and cache APIs, but the attention and decoder layers are replaced by Q-Project implementations.
Architecture
Input token IDs
│
▼
Tied token embedding (32,768 × 512)
│
▼
┌──────────────────────────────────────────┐
│ QDecoderLayer × 10 │
│ │
│ RMSNorm │
│ │ │
│ ▼ │
│ GQA: 8 query heads / 2 KV heads │
│ ├─ per-head QK-Norm │
│ ├─ RoPE positional encoding │
│ └─ content-dependent scalar output gate │
│ │ │
│ └──────────── residual connection │
│ │
│ RMSNorm │
│ │ │
│ ▼ │
│ SwiGLU MLP (512 → 1,792 → 512) │
│ └─ content-dependent scalar output gate │
│ │ │
│ └──────────── residual connection │
└──────────────────────────────────────────┘
│
▼
Final RMSNorm
│
▼
Tied language-model head → next-token logits
Configuration
| Parameter | Value |
|---|---|
| Total parameters | 50,878,208 |
| Vocabulary size | 32,768 |
| Hidden size | 512 |
| Decoder layers | 10 |
| Query attention heads | 8 |
| Key/value heads | 2 |
| Attention head dimension | 64 |
| MLP intermediate size | 1,792 |
| Maximum context length | 2,048 tokens |
| Positional encoding | RoPE in every layer |
| RoPE theta | 10,000 |
| Normalization | RMSNorm, epsilon 1e-5 |
| Attention stabilization | Per-head QK-Norm |
| MLP | SwiGLU / SiLU |
| Residual branch gates | Scalar, content-dependent, attention + MLP |
| Embedding / LM head | Tied |
| Attention dropout | 0.0 |
| Training attention backend | PyTorch SDPA |
The implementation supports selective NoPE layers for architecture experiments. This checkpoint sets nope_every_n=None, so RoPE is active in all ten layers.
Why this architecture?
A 512-wide, 10-layer decoder
At roughly 50M parameters, allocating capacity is a trade-off between vocabulary embeddings, depth, width, and MLP size. Ten 512-dimensional layers give the model enough sequential depth to build progressively richer representations without making every attention and MLP operation too expensive. The 1,792-dimensional SwiGLU MLP provides substantial nonlinear capacity while keeping the full model within the target size.
Tied input and output embeddings
With a 32K vocabulary and a hidden size of 512, one embedding matrix contains about 16.8M parameters. Sharing it with the output language-model head avoids a second matrix of the same size. This is especially important in a 50M model: the saved parameters can be used in the decoder layers instead of duplicating lexical storage.
Grouped-query attention
The model has 8 query heads but only 2 key/value heads. Four query heads share each key/value head. Compared with standard multi-head attention using 8 independent KV heads, this reduces the size of the KV cache by approximately 4× while preserving multiple query subspaces.
This design was chosen primarily for lower autoregressive inference memory and better generation throughput. The trade-off is reduced key/value diversity compared with full multi-head attention.
Per-head QK-Norm
Queries and keys are independently RMS-normalized inside each attention head before attention scores are computed. This limits uncontrolled growth in their magnitudes, makes attention logits less sensitive to activation scale, and improves optimization stability during long pretraining runs.
QK-Norm adds very few parameters: only one scale vector for queries and one for keys in each layer.
Content-dependent scalar residual gates
Both attention and MLP outputs are modulated before they are added to the residual stream:
gate(x) = 2 × sigmoid(Wx)
output = gate(x) × branch(x)
The gate produces one scalar per token rather than a full hidden-size vector. It therefore lets the model strengthen or suppress an entire residual branch based on the current token representation with minimal parameter and compute overhead.
The multiplier of 2 centers the gate near 1 at initialization, so each gated branch begins close to an ordinary Transformer branch instead of being nearly closed. The model can then learn token-dependent deviations during training.
Mistral-compatible foundation
Q-50M reuses the surrounding Hugging Face Mistral model interfaces, RMSNorm layout, SwiGLU MLP convention, generation utilities, and cache behavior. This reduces implementation risk and makes the custom architecture easier to use with the Transformers ecosystem. The novel parts remain localized in QAttention, QMLP, QDecoderLayer, and QConfig.
Pretraining
| Setting | Value |
|---|---|
| Dataset | FineWeb-Edu, sample-10BT |
| Training tokens | 5,000,003,584 (~5B) |
| Training steps | 152,588 |
| Sequence length | 2,048 |
| Micro-batch size | 2 sequences |
| Gradient accumulation | 8 |
| Effective tokens per optimizer step | 32,768 |
| Optimizer | Fused AdamW |
| Peak learning rate | 3e-4 |
| Adam betas | (0.9, 0.95) |
| Weight decay | 0.1 |
| LR schedule | Warmup-stable-decay; 1% warmup, final 10% linear decay |
| Precision | FP16 |
| Compilation | torch.compile |
| Training hardware | 1× NVIDIA Tesla V100 16GB |
| Random seed | 2026 |
Tokenizer
The model uses a 32,768-entry byte-level BPE tokenizer trained from scratch on FineWeb-Edu text. Byte-level fallback allows arbitrary UTF-8 text to be represented without unknown-character failures, although the model itself was trained primarily on English data.
Evaluation
The final checkpoint was evaluated on a fixed FineWeb-Edu validation cache containing 524,288 tokens.
| Metric | Value |
|---|---|
| Final training loss | 3.5127 |
| Validation loss | 3.1974 |
| Validation perplexity | 24.47 |
An observed greedy-generation speed on a Tesla V100 was approximately 36.6 tokens/s. This is a local smoke-test measurement, not a standardized benchmark; speed depends on prompt length, generated length, precision, backend, and hardware.
Perplexity should only be compared directly with models evaluated using the same tokenizer, tokenization pipeline, context construction, and validation data.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "q-project/Q-50M-Base"
tokenizer = AutoTokenizer.from_pretrained(
model_id,
trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
torch_dtype=torch.float16,
device_map="auto",
)
model.eval()
prompt = "Artificial intelligence can help people by"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
output = model.generate(
**inputs,
max_new_tokens=128,
do_sample=True,
temperature=0.8,
top_p=0.95,
top_k=50,
repetition_penalty=1.1,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
trust_remote_code=True is required because the repository includes the custom QConfig and QForCausalLM implementations.
Base-model prompting
Use a natural text prefix and let the model continue it. Do not apply a chat template to this checkpoint. For instruction following, use a separately released Q-50M-Instruct checkpoint.
Intended use
Q-50M-Base is intended for:
- research on compact language-model architectures;
- experimentation with QK-Norm and lightweight residual gating;
- causal text continuation;
- educational use and local inference;
- supervised fine-tuning, instruction tuning, or domain adaptation.
It is not intended to be used as a factual authority, safety-critical system, autonomous agent, or production chat assistant without additional evaluation and alignment work.
Limitations
- At 50.9M parameters, the model has limited factual knowledge, reasoning depth, and long-range coherence.
- The pretraining corpus is primarily English; quality in other languages is not established.
- The model can produce incorrect, biased, repetitive, unsafe, or nonsensical text.
- This base checkpoint was not instruction-tuned and may ignore questions or commands.
- A 2,048-token configured context does not imply uniform quality across the entire window.
- FineWeb-Edu is derived from web data and may contain residual errors or undesirable content despite filtering.
- The architecture and evaluation should be treated as an experimental research release.
Users are responsible for evaluating outputs and adding safeguards appropriate to their application.
Repository files required for release
Because this is a custom Transformers architecture, the model repository should include:
README.md
LICENSE
Q_Logo.svg
config.json
configuration_q.py
modeling_q.py
generation_config.json
model.safetensors
tokenizer.json
tokenizer_config.json
special_tokens_map.json
config.json must expose the custom classes through auto_map, for example:
{
"auto_map": {
"AutoConfig": "configuration_q.QConfig",
"AutoModel": "modeling_q.QModel",
"AutoModelForCausalLM": "modeling_q.QForCausalLM"
}
}
Citation
@misc{q50mbase2026,
title = {Q-50M-Base: A Compact Gated Decoder Language Model},
author = {Nikolay Kompanets},
year = {2026},
url = {https://huggingface.co/q-project/Q-50M-Base}
}
Acknowledgements
Q-50M-Base uses the Hugging Face Transformers ecosystem and was pretrained on FineWeb-Edu. The model implementation derives its standard decoder components and public interfaces from the Transformers Mistral implementation, with custom Q-Project attention normalization and residual gating modules.
- Downloads last month
- 16