slm-pipeline-151m

A 151M-parameter language model trained from scratch through every stage of the modern pipeline — pre-training → mid-training → SFT → DPO → RLVR/GRPO — on one H100 in 20 hours 20 minutes. Every stage is plain PyTorch: no TRL, no Trainer, every loss written out.

This is a demonstration of a training pipeline, not a useful assistant. Read the evaluation section before doing anything with it.

⚠️ This is not a transformers model

config.json declares "architecture": "slm-pipeline/Transformer". It is not a transformers architecture and AutoModelForCausalLM.from_pretrained will not load it. The weights are plain safetensors for the custom model defined in slm/model.py of the producing repo.

from huggingface_hub import snapshot_download
from safetensors.torch import load_file
import torch, json

path = snapshot_download("alpharomercoma/slm-pipeline-151m")
cfg = json.load(open(f"{path}/config.json"))
state = load_file(f"{path}/model.safetensors")

# Requires slm-pipeline on your PYTHONPATH:
from slm.model import ModelConfig, Transformer
model = Transformer(ModelConfig.from_dict(cfg))
model.load_state_dict(state)
model.eval()

The tokenizer is a byte-level BPE trained on this corpus and loads with tokenizers.Tokenizer.from_file("tokenizer.json"). Chat turns use single reserved control tokens rather than ChatML text: <|bos|> <|user|> … <|end|> <|assistant|> … <|end|> <|eos|>.

Evaluation — read this before using it

Measured on the final 6_rlvr_grpo checkpoint, with the chat template.

Benchmark	Score	Random	n
HellaSwag	37.3%	25%	1000
ARC-Easy	47.7%	25%	570
MMLU	23.8%	25%	1000
GSM8K	1.5%	0%	200
Perplexity (held-out FineWeb-Edu)	25.10	—	—

MMLU is at chance and GSM8K is at the floor. HellaSwag and ARC-Easy are meaningfully above random, so the model has real commonsense signal — but at 151M parameters it has very little knowledge, cannot do arithmetic, and hallucinates freely. It should not be used for anything consequential.

Capability across the whole pipeline (single run, one seed, aggregate scores only — every multiple-choice movement between stages is inside one standard error, so read these as "nothing was lost", not as stage-by-stage gains):

Stage	ppl	HellaSwag	ARC-e	MMLU	GSM8K	#### format rate
2 pre-training	20.85	38.5%	45.8%	24.0%	1.0%	0.000
3 mid-training	19.70	37.3%	45.1%	26.6%	4.0%	0.900
4 SFT	25.11	37.4%	48.1%	23.8%	1.0%	0.055
5 DPO	25.07	37.3%	47.9%	24.3%	2.5%	0.060
6 RLVR/GRPO	25.10	37.3%	47.7%	23.8%	1.5%	0.985

The perplexity rise at SFT is not a regression — it is measured on held-out web text while the model is being specialised for dialogue.

The one large, unambiguous signal is the format-rate column, and it tells a three-stage story: mid-training taught the GSM8K #### <answer> convention (0.900), SFT destroyed it (0.055, because SmolTalk contains #### in 2,733 of 455,483 conversations), and GRPO recovered it (0.985). GRPO recovered a behaviour rather than discovering one, and it did not improve arithmetic: group-relative advantages give no gradient once every rollout in a group scores identically, which is what happens when all of them are well-formed and none correct.

Architecture

Parameters	151,028,992 (125.9M non-embedding)
Layers	20
d_model	768
Attention	12 query heads / 4 KV heads (GQA)
FFN	SwiGLU, hidden 2048
Normalisation	RMSNorm pre-norm, with QK-norm
Positions	RoPE, θ = 100,000
Context	4096 (pre-trained at 2048, extended during mid-training)
Vocabulary	32,768 byte-level BPE, trained on this corpus
Embeddings	tied
Biases	none

Training

Stage	Data	Tokens / steps	Wall clock
Pre-training	FineWeb-Edu	8.26B tokens, 8000 steps	7h 16m
Mid-training	FineMath / code / MMLU-aux / GSM8K / SmolTalk	1.68B tokens, 1600 steps	1h 45m
SFT	smol-smoltalk	449,023 conversations × 2 epochs	3h 47m
DPO	UltraFeedback-binarized	52,951 pairs	22m
RLVR/GRPO	GSM8K (train split)	400 steps × 64 rollouts	30m

Pre-training consumed 8.26B tokens from a 9.89B-token corpus — 0.84 epochs, so no token was seen twice. That is 54.7 tokens per parameter, deliberately far past Chinchilla-optimal (~3B), which is the right trade when you care about quality at a fixed inference cost.

Pre-training and mid-training are two halves of one Warmup–Stable–Decay schedule: stage 2 holds the LR flat at 6.0e-4 and ends hot; stage 3 picks it up there and anneals to exactly 0.0 on a better mixture. Peak MFU 0.412; pre-training held 317,600 tok/s at MFU 0.364 in steady state.

GSM8K train/test splits are disjoint by construction: gsm8k_text (mid-training) and the GRPO rollout prompts both come from split="train" (7,473 rows); evaluation uses split="test" (1,319 rows). No further decontamination against the web corpora was done.

Provenance

• Producing code: slm-pipeline at commit c71c1fe
• All 28 source files verified identical between the training host and local HEAD by md5, mid-run
• Hardware: 1× NVIDIA H100 80GB SXM, Nebius eu-north1, driver 550.163.01, torch 2.13.0+cu129
• evals/ in this repo contains the raw per-stage evaluation JSON and the generated progression table

Limitations

One run, one seed, one checkpoint per stage, scored once. No replicates, no confidence intervals on differences, and the evaluation harness stored aggregate accuracies only — so no paired testing is possible after the fact. Treat every between-stage difference here as un-established in either direction.

License

Apache-2.0 for the weights. Training data carries its own licences — FineWeb-Edu (ODC-By), FineMath, smol-smoltalk, UltraFeedback, GSM8K (MIT), MMLU auxiliary train — and you are responsible for complying with those if you build on this.