NBR-1B: Brazilian Portuguese Language Model
NBR-1B is a 1.13 billion parameter language model trained from scratch for Brazilian Portuguese.
Model Details
| Attribute | Value |
|---|---|
| Parameters | 1.13B |
| Architecture | LLaMA-style (GQA, RMSNorm, SwiGLU, RoPE) |
| Hidden Size | 2048 |
| Layers | 24 |
| Attention Heads | 16 |
| KV Heads | 4 |
| Vocabulary | 32,000 (BPE) |
| Context Length | 2048 |
| Training Tokens | 3.12B |
| Final Loss | ~2.8 |
Training Data
- CulturaX PT (40%)
- FineWeb-2 PT (52%)
- mC4 PT (5%)
- CC-News PT v2 (2%)
- Books PT (1%)
Usage
This is a base model for text completion. Use with transformers library.
License
Apache 2.0
- Downloads last month
- 2