MihaiPopa-1/GPT-3-Tiny-Exp-52M

GPT-3 Tiny — 52M Parameter Language Model

A faithful ~52M parameter replica of the GPT-3 architecture (Brown et al., 2020), trained from scratch on a mixture of TinyStories, FineWeb Edu, and a full scrape of the Minecraft Wiki captured on June 5, 2026 — eleven days before the Chaos Cubed drop (Java Edition 26.2) released publicly.

---

Model Description

GPT-3 Tiny replicates every architectural decision from the GPT-3 paper at a reduced scale:

Hyperparameter	This model	GPT-3 Small (125M)	Source
n_layers	8	12	Table 2.1
d_model	512	768	Table 2.1
n_heads	8	12	Table 2.1
d_head	64	64	d_model / n_heads
d_ff	2048	3072	4 × d_model
n_ctx	2048	2048	Section 2.1
n_vocab	50,257	50,257	GPT-2 BPE
Total params	~52M	125M	—

Architectural choices (all matching GPT-3):

• Decoder-only Transformer (no cross-attention)
• Pre-LayerNorm (LayerNorm before each sub-block, as in GPT-2)
• Learned positional embeddings
• Multi-head causal (masked) self-attention
• FFN: Linear → GeLU → Linear, hidden = 4 × d_model
• Tied input/output embedding weights
• Final LayerNorm after last block

---

Training

Hardware: Google Colab T4 GPU (16 GB VRAM)
Precision: fp16 mixed precision
Optimizer: AdamW (β₁=0.9, β₂=0.95, ε=1e-8)
Learning rate: 6×10⁻⁴ with cosine decay and linear warmup
Weight decay: 0.1 (weight matrices only; biases and LayerNorm excluded)
Gradient clipping: 1.0
Batch size: 32 sequences × 512 tokens
Tokenizer: GPT-2 BPE via tiktoken (50,257 vocab)

Training Data

Dataset	Description	Approx. size (of slice)
TinyStories	Simple children's stories	~180 MB
FineWeb Edu	Filtered educational web text	~180 MB
Minecraft Wiki	Full scrape, June 5, 2026	~75 MB (after some cleaning)

The Minecraft Wiki scrape is the most distinctive component. It was captured eleven days before the Chaos Cubed game drop (Java Edition 26.2, released June 16, 2026), and therefore contains snapshot history pages, block/item tables, and feature descriptions for sulfur cubes, sulfur caves, cinnabar blocks, and the new 26.x versioning format — content that did not exist in any prior public model's training data.

Known data issue: No <|endoftext|> separator token was inserted between documents during packing. This causes cross-domain contamination in generation (see Limitations).

---

Intended Uses

• Educational: Understanding GPT-3 architecture behavior at small scale
• Research: Studying hallucination patterns, recency underrepresentation, and cross-domain bleed in small LMs
• Minecraft community: Exploring model knowledge of pre-release Chaos Cubed content
• Experimentation: Fine-tuning on domain-specific corpora

Out-of-scope uses

This model is not suitable for factual question answering, any production application, or any task requiring reliable knowledge retrieval. It hallucinates fluently and confidently.

---

Evaluation & Observed Behavior

The model was evaluated qualitatively through extensive prompted generation on Minecraft Wiki-style inputs. A detailed hallucination taxonomy was identified:

What the model gets right

• MediaWiki template syntax: {{HistoryLine}}, {{BlockLink}}, {{ItemLink}}, [[File:...|32px]], |- table rows — all generated correctly and consistently
• Historical snapshot knowledge: Snapshot names (15w47a, 18w10a, 19w11a), The Flattening (1.13), pre-flattening numeric block IDs
• New versioning format: Correctly generates dev=26.2, exp=Drop 2 of 2026, dev=snap1 through dev=snap6 — the new year.drop format introduced in December 2025
• Chaos Cubed content: Correctly recalls sulfur cubes, sulfur caves, cinnabar blocks, and the minecraft:sulfur_cube_archetype behavioral system (including the real archetype name regular)

Best observed output:

|{{HistoryLine|||dev=15w47a|Prior to [[Java Edition 1.13/Flattening|''The Flattening'']], 
this item's numeral ID was 69.}}

Correct snapshot, correct historical event, correct wiki link format, correct numeric ID for Lever.

Hallucination taxonomy

Type	Example	Root cause
Temporal detachment	Pale Moss Carpet in 19w06a (correct block, wrong era by 5 years)	Spurious co-occurrence in training data
Stable false belief	23w10a generated repeatedly as real snapshot (doesn't exist)	Burned-in wrong association, fires reliably
Recency underrepresentation	Sulfur cube content correctly known but sometimes attributed to fake 23w10a instead of real 26.2	YYwNNa format has thousands of training pages; 26.x has very few
Cross-edition contamination	Bedrock Preview 1.21.50.20 version string inside === ''Java Edition'' === section	Attention over section header fades mid-generation
Domain bleed	FineWeb Edu academic prose mid-table: "The majority of the population in the world is the number of population."	No `<
TinyStories eruption	Children's story dialogue ("I'm sorry, Lily," she says.) inserted between {{HistoryLine}} entries	Same cause as domain bleed
Repetition collapse	Added cauldrons, which can be used to create cauldron and cauldrons.	Uncertain completion → copy last noun
Token budget corruption	ofrimson (of + Crimson merged), carpetes	Attention consumed by long image filename tokens
Category confusion	Cake described as a status effect	Cake and effects share (inventory) texture filename pattern
Mechanic shape without content	sulfur cubes absorb [netherrack / polished redstone / coral / light / durability]	Correct mechanic template, sampled wrong target each time

The sulfur cube finding

Across more than eight independent generations, the model consistently produced sulfur cubes with correct metadata (dev=26.2, exp=Drop 2 of 2026) and partially correct mechanics (the regular archetype is a real entry in the minecraft:sulfur_cube_archetype registry). The model incorrectly placed sulfur caves in the Nether rather than the Overworld — likely because sulfur is Nether-associated in thousands of Minecraft mod pages that predated the official feature. This is not hallucination; it is accurate knowledge with one wrong spatial association.

The model's consistent generation of sulfur cube content before the drop's public release is a concrete example of recency underrepresentation: the model has the knowledge, but the old YYwNNa snapshot format is so statistically dominant that it sometimes overwrites the correct 26.2 attribution.

---

Limitations

• Hallucinates confidently and fluently. All outputs should be treated as fiction unless independently verified.
• No document separators in training. Cross-domain bleed (Minecraft Wiki → FineWeb Edu → TinyStories mid-generation) occurs regularly and is not fixable without retraining with <|endoftext|> separators.
• 52M parameters is very small. Coherent factual recall requires ~10× more parameters at minimum.
• No instruction tuning or RLHF. This is a raw language model; it does not follow instructions.
• Biome placement errors for new content. Sulfur caves are Overworld; model places them in the Nether.
• Fake snapshot names. 23w10a, 19w50a, 24w40a, and others are generated with high confidence but do not exist.

---

Citation

If you use this model or reference the hallucination analysis in research, please cite:

@misc{popa2026gpt3tiny,
  author    = {Popa, Mihai},
  title     = {GPT-3 Tiny: A 52M Parameter GPT-3 Replica Trained on the Minecraft Wiki},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/MihaiPopa-1/GPT-3-Tiny-Exp-52M}
}

---

Model Card Contact

Mihai Popa — via Hugging Face repository issues.