TR-HASH MoE 200M

PIQA-selected full-SFT checkpoint

Explore how two deterministic, layer-specific token-ID hashes select a top-2 pair of residual experts while a shared dense SwiGLU path preserves contextual computation.

201.2M parameters ≈162B source-token exposure 16 layers · multi-hash top-2

Interactive companion — not additional experimental evidence.

Token

Contextual hidden state x_t The same contextual vector enters both branches

Token identifier t routing · tokenizer ID 22447

Shared SwiGLU Width 3,072 · all tokens · contextual computation

Multi-hash route table r_l,1:2(t) Two deterministic hashes per layer · two of four experts

Residual experts · width 256 each E0 E1 E2 E3

Feed-forward output Shared + 2 × mean(E0, E2)

Layer-specific deterministic routes

Checkpoint-derived tokenizer IDs and deterministic two-hash top-2 assignments.

E0E1E2E3 Layer 10.5—0.5— Layer 4—0.50.5— Layer 8—0.50.5— Layer 12——0.50.5 Layer 16——0.50.5

Token “routing” follows the persisted route table of the final checkpoint. Every occurrence keeps the same expert pair at a given layer, while its hidden-state input still changes with context.

Recorded training diagnostics

Source and scheduled SFT evaluations · NLL, lower is better

1.7628

1.2209

Held-out SFT NLL

68.39%

69.31%

PIQA acc_norm

Source SFT measurement

Measurement	Source	SFT checkpoint
Held-out SFT NLL	1.7628 @ source	1.2209 @ epoch 3
Held-out SFT PPL	5.83	3.39
PIQA accuracy	68.66% refinement	68.82% epoch 2
PIQA acc_norm	68.39% refinement	69.31% epoch 2
SFT coverage	209k records	3 epochs · 238.9M tokens

SFT loss uses the fixed 2,100-example held-out split. PIQA uses all 1,838 validation examples, zero-shot causal continuation likelihood, no chat template, and FP16 eager evaluation.

Prompt

TR-HASH MoE 200M · Full SFT ready

The model response will stream here token by token.

Decode configuration

Chat endpoint · temperature 0.4 · top-k 30 · top-p 0.85 · repetition penalty 1.1 · maximum 384 new tokens.

This is a qualitative demonstration of the released SFT checkpoint, not an evaluation result.

TR-HASH MoE 200M · live qualitative generation