Upload folder using huggingface_hub
Browse files- README.md +28 -7
- adapter_config.json +2 -2
- adapter_model.safetensors +1 -1
- choice_bank_fit.json +7 -7
- choice_head.json +2 -2
- confidence_calibration.json +0 -0
- state_prototypes.json +0 -0
- temperature_calibration.json +61 -61
README.md
CHANGED
|
@@ -27,8 +27,26 @@ pipeline_tag: text-classification
|
|
| 27 |
> [`munod/tachyone-multi`](https://huggingface.co/munod/tachyone-multi)); measured numbers below come
|
| 28 |
> from `benchmarks/report.md`.
|
| 29 |
>
|
| 30 |
-
> **This revision is B-
|
| 31 |
-
>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
> (sampler strides decoupled so every language trains every label, evals made
|
| 33 |
> leave-one-template-out, phrase content ×3, volume 1.7×) and every training set regenerated so
|
| 34 |
> training and the holdout eval share **0%** of templates. The multilingual lineage carries the
|
|
@@ -142,9 +160,9 @@ interleaved touch-up (`training/interleave_continue.py`).
|
|
| 142 |
| English (ModernBERT-large + five-domain LoRA r=16 + choice-head bank) | support | **1.0000** | 0.188 | 57.7 |
|
| 143 |
| English, five-domain split | 5 domains | **0.9983** | 0.191 | 57.7 |
|
| 144 |
| English, **holdout phrasing** (never in training) | 5 domains | **0.9825** | 0.188 | 58.0 |
|
| 145 |
-
| Multilingual (mmBERT-base + LoRA r=64 + fitted bank) — routed runtime | support | **0.
|
| 146 |
-
| Multilingual, five-domain split | 5 domains | **0.
|
| 147 |
-
| Multilingual, **holdout phrasing** (never in training) | 5 domains | **0.
|
| 148 |
|
| 149 |
The English ECE column reads the **pooled never-trained fit** (below): `choice` is flattened to
|
| 150 |
T=10.0 to calibrate held-out rows, which shows up honestly in-domain (0.5243 `choice` ECE —
|
|
@@ -214,8 +232,11 @@ rows fall through to the English checkpoint by language routing, `.specs/project
|
|
| 214 |
0.05 ECE target on that split, and on the **holdout** the B-1 per-language ECE criterion was
|
| 215 |
**NOT met** at B-15 (7 cells, worst `score/it` 0.1495) — the **B-16** cycle (serve-key
|
| 216 |
detector + fitted confidence, 2026-10-09) moved it to **11/12 with `score/it` 0.0521
|
| 217 |
-
recorded NOT met**
|
| 218 |
-
|
|
|
|
|
|
|
|
|
|
| 219 |
The CUDA-graph fast path
|
| 220 |
(`TACHYONE_FAST=1`) gives **2.53×** p50 on English (17.31 → 6.83 ms) and **3.71×** on the
|
| 221 |
multilingual five-domain path (14.23 → 3.84 ms) — re-measured on the v0.9.0 artifacts
|
|
|
|
| 27 |
> [`munod/tachyone-multi`](https://huggingface.co/munod/tachyone-multi)); measured numbers below come
|
| 28 |
> from `benchmarks/report.md`.
|
| 29 |
>
|
| 30 |
+
> **This revision is B-17: the teacher expanded the tone banks, the multilingual arm was
|
| 31 |
+
> retrained, and B-1's per-language ECE acceptance closes.** **352 phrases** authored by a
|
| 32 |
+
> local `qwen3.6:35b` teacher went into every `neutral`/`request` bank (five domains × six
|
| 33 |
+
> languages, inserted **before** the held-out phrase — the holdout evals are byte-identical,
|
| 34 |
+
> sha-pinned); the multilingual checkpoint retrained with the unchanged B-15 recipe and
|
| 35 |
+
> **English was not touched** (byte-identical numbers). On phrasing never seen in any
|
| 36 |
+
> training: **multilingual 0.9264 (routed), English 0.9825** — the in-template → holdout gap
|
| 37 |
+
> collapsed **−0.1105 → −0.0165** — and **all 12 `(primitive, language)` cells now sit at
|
| 38 |
+
> ECE ≤ 0.05** (worst `score/fr` 0.0257; `score/it` **0.0521 → 0.0014**, accuracy
|
| 39 |
+
> 0.8699 → 0.9928) — **B-1 criterion 3 (NFR-C06) CLOSED** on the frozen yardstick. Also
|
| 40 |
+
> improved: `choice` holdout 0.9248 → 0.9808, the `noul` tripwire 0.7807 → 0.9735, every
|
| 41 |
+
> language ≥ 0.9928 (`es` 1.0000), in-template multilingual ECE 0.0228 → 0.0036 (zero gate
|
| 42 |
+
> exceptions everywhere). Disclosures, never smoothed: the expansion targeted a diagnosed
|
| 43 |
+
> mechanism (BACKLOG **B-17** carries the per-domain table); off-domain **MASSIVE
|
| 44 |
+
> 0.074 → 0.058 with `ECE raw` 0.046 → 0.197** — in-domain gains re-sharpened off-domain
|
| 45 |
+
> confidence (the L-007/L-015 trade), still 3.4× its chance; and the English P3 `choice`
|
| 46 |
+
> ECE **0.5243** (pooled fit at T = 10.0) stands exactly as previously disclosed.
|
| 47 |
+
>
|
| 48 |
+
> **The v0.9.0/v0.10.0 revisions were B-15/B-16 (history).** B-15 retrained both checkpoints
|
| 49 |
+
> on `template_split: "train"` after the data recipe was corrected end to end
|
| 50 |
> (sampler strides decoupled so every language trains every label, evals made
|
| 51 |
> leave-one-template-out, phrase content ×3, volume 1.7×) and every training set regenerated so
|
| 52 |
> training and the holdout eval share **0%** of templates. The multilingual lineage carries the
|
|
|
|
| 160 |
| English (ModernBERT-large + five-domain LoRA r=16 + choice-head bank) | support | **1.0000** | 0.188 | 57.7 |
|
| 161 |
| English, five-domain split | 5 domains | **0.9983** | 0.191 | 57.7 |
|
| 162 |
| English, **holdout phrasing** (never in training) | 5 domains | **0.9825** | 0.188 | 58.0 |
|
| 163 |
+
| Multilingual (mmBERT-base + LoRA r=64 + fitted bank) — routed runtime | support | **0.9300** | 0.038 | 52.8 |
|
| 164 |
+
| Multilingual, five-domain split | 5 domains | **0.9221** | 0.026 | 53.5 |
|
| 165 |
+
| Multilingual, **holdout phrasing** (never in training) | 5 domains | **0.9264** | 0.032 | 53.3 |
|
| 166 |
|
| 167 |
The English ECE column reads the **pooled never-trained fit** (below): `choice` is flattened to
|
| 168 |
T=10.0 to calibrate held-out rows, which shows up honestly in-domain (0.5243 `choice` ECE —
|
|
|
|
| 232 |
0.05 ECE target on that split, and on the **holdout** the B-1 per-language ECE criterion was
|
| 233 |
**NOT met** at B-15 (7 cells, worst `score/it` 0.1495) — the **B-16** cycle (serve-key
|
| 234 |
detector + fitted confidence, 2026-10-09) moved it to **11/12 with `score/it` 0.0521
|
| 235 |
+
recorded NOT met** — and the **B-17** cycle (teacher bank expansion + retrain, 2026-10-10)
|
| 236 |
+
**closed it**: **all 12/12 cells ≤ 0.05 on the same frozen holdout** (worst `score/fr`
|
| 237 |
+
0.0257, `score/it` **0.0014** at accuracy 0.9928), `choice` 0.9248 → 0.9808 and every other
|
| 238 |
+
axis improved or byte-identical. **NFR-C06 met**; the trail is in BACKLOG
|
| 239 |
+
**B-1**/**B-15**/**B-16**/**B-17**.
|
| 240 |
The CUDA-graph fast path
|
| 241 |
(`TACHYONE_FAST=1`) gives **2.53×** p50 on English (17.31 → 6.83 ms) and **3.71×** on the
|
| 242 |
multilingual five-domain path (14.23 → 3.84 ms) — re-measured on the v0.9.0 artifacts
|
adapter_config.json
CHANGED
|
@@ -35,9 +35,9 @@
|
|
| 35 |
"rank_pattern": {},
|
| 36 |
"revision": null,
|
| 37 |
"target_modules": [
|
| 38 |
-
"Wo",
|
| 39 |
"Wqkv",
|
| 40 |
-
"Wi"
|
|
|
|
| 41 |
],
|
| 42 |
"target_parameters": null,
|
| 43 |
"task_type": null,
|
|
|
|
| 35 |
"rank_pattern": {},
|
| 36 |
"revision": null,
|
| 37 |
"target_modules": [
|
|
|
|
| 38 |
"Wqkv",
|
| 39 |
+
"Wi",
|
| 40 |
+
"Wo"
|
| 41 |
],
|
| 42 |
"target_parameters": null,
|
| 43 |
"task_type": null,
|
adapter_model.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 54088096
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:eb946aab6a74a3e5cbf1811c1c4cf85579941d1b56e9c9208da188848852b616
|
| 3 |
size 54088096
|
choice_bank_fit.json
CHANGED
|
@@ -1,10 +1,10 @@
|
|
| 1 |
{
|
| 2 |
"arm": "bank",
|
| 3 |
"config": {
|
| 4 |
-
"bank_dir": "checkpoints/
|
| 5 |
"batch_size": 64,
|
| 6 |
"choice_rank": 128,
|
| 7 |
-
"control_dir": "checkpoints/
|
| 8 |
"data_path": "data/train_multi_domains.jsonl",
|
| 9 |
"device": "auto",
|
| 10 |
"epochs": 8,
|
|
@@ -13,7 +13,7 @@
|
|
| 13 |
"max_records": null,
|
| 14 |
"model_id": "jhu-clsp/mmBERT-base",
|
| 15 |
"seed": 42,
|
| 16 |
-
"trunk_adapter": "checkpoints/
|
| 17 |
"val_split": 0.1
|
| 18 |
},
|
| 19 |
"domains": [
|
|
@@ -26,10 +26,10 @@
|
|
| 26 |
"fit": {
|
| 27 |
"arm": "bank",
|
| 28 |
"epochs": 8,
|
| 29 |
-
"temperature": 0.
|
| 30 |
-
"train_loss": 0.
|
| 31 |
"train_records": 15660,
|
| 32 |
-
"val_loss": 0.
|
| 33 |
"val_records": 1740
|
| 34 |
},
|
| 35 |
"languages": [
|
|
@@ -40,5 +40,5 @@
|
|
| 40 |
"nl",
|
| 41 |
"pt"
|
| 42 |
],
|
| 43 |
-
"trunk_adapter": "checkpoints/
|
| 44 |
}
|
|
|
|
| 1 |
{
|
| 2 |
"arm": "bank",
|
| 3 |
"config": {
|
| 4 |
+
"bank_dir": "checkpoints/multi_b17_il_fit_bank",
|
| 5 |
"batch_size": 64,
|
| 6 |
"choice_rank": 128,
|
| 7 |
+
"control_dir": "checkpoints/multi_b17_il_fit_ctrl",
|
| 8 |
"data_path": "data/train_multi_domains.jsonl",
|
| 9 |
"device": "auto",
|
| 10 |
"epochs": 8,
|
|
|
|
| 13 |
"max_records": null,
|
| 14 |
"model_id": "jhu-clsp/mmBERT-base",
|
| 15 |
"seed": 42,
|
| 16 |
+
"trunk_adapter": "checkpoints/multi_b17_il",
|
| 17 |
"val_split": 0.1
|
| 18 |
},
|
| 19 |
"domains": [
|
|
|
|
| 26 |
"fit": {
|
| 27 |
"arm": "bank",
|
| 28 |
"epochs": 8,
|
| 29 |
+
"temperature": 0.9909090399742126,
|
| 30 |
+
"train_loss": 0.013521413194443138,
|
| 31 |
"train_records": 15660,
|
| 32 |
+
"val_loss": 0.020827124596054517,
|
| 33 |
"val_records": 1740
|
| 34 |
},
|
| 35 |
"languages": [
|
|
|
|
| 40 |
"nl",
|
| 41 |
"pt"
|
| 42 |
],
|
| 43 |
+
"trunk_adapter": "checkpoints/multi_b17_il"
|
| 44 |
}
|
choice_head.json
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2c7222ea220d28da546c067b10ab1bf343d5ff26d0c7fe1db08984ea9f21ac76
|
| 3 |
+
size 39897981
|
confidence_calibration.json
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
state_prototypes.json
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
temperature_calibration.json
CHANGED
|
@@ -4,134 +4,134 @@
|
|
| 4 |
"choice": {
|
| 5 |
"by_language": {
|
| 6 |
"de": {
|
| 7 |
-
"ece_after": 0.
|
| 8 |
-
"ece_before": 0.
|
| 9 |
"n": 830,
|
| 10 |
-
"temperature":
|
| 11 |
},
|
| 12 |
"es": {
|
| 13 |
-
"ece_after": 0.
|
| 14 |
-
"ece_before":
|
| 15 |
"n": 840,
|
| 16 |
-
"temperature":
|
| 17 |
},
|
| 18 |
"fr": {
|
| 19 |
-
"ece_after": 0.
|
| 20 |
-
"ece_before": 0.
|
| 21 |
"n": 830,
|
| 22 |
-
"temperature":
|
| 23 |
},
|
| 24 |
"it": {
|
| 25 |
-
"ece_after": 0.
|
| 26 |
-
"ece_before": 0.
|
| 27 |
"n": 830,
|
| 28 |
-
"temperature":
|
| 29 |
},
|
| 30 |
"nl": {
|
| 31 |
-
"ece_after": 0.
|
| 32 |
-
"ece_before": 0.
|
| 33 |
"n": 830,
|
| 34 |
-
"temperature":
|
| 35 |
},
|
| 36 |
"pt": {
|
| 37 |
-
"ece_after": 0.
|
| 38 |
-
"ece_before": 0.
|
| 39 |
"n": 840,
|
| 40 |
-
"temperature":
|
| 41 |
}
|
| 42 |
},
|
| 43 |
-
"ece_after": 0.
|
| 44 |
-
"ece_before": 0.
|
| 45 |
"n": 5000,
|
| 46 |
-
"temperature":
|
| 47 |
},
|
| 48 |
"noul": {
|
| 49 |
"by_language": {
|
| 50 |
"de": {
|
| 51 |
-
"ece_after": 0.
|
| 52 |
-
"ece_before": 0.
|
| 53 |
"n": 830,
|
| 54 |
-
"temperature": 0.
|
| 55 |
},
|
| 56 |
"es": {
|
| 57 |
-
"ece_after":
|
| 58 |
-
"ece_before": 0.
|
| 59 |
"n": 840,
|
| 60 |
-
"temperature": 0.
|
| 61 |
},
|
| 62 |
"fr": {
|
| 63 |
-
"ece_after": 0.
|
| 64 |
-
"ece_before": 0.
|
| 65 |
"n": 830,
|
| 66 |
-
"temperature": 0.
|
| 67 |
},
|
| 68 |
"it": {
|
| 69 |
-
"ece_after": 0.
|
| 70 |
-
"ece_before": 0.
|
| 71 |
"n": 830,
|
| 72 |
-
"temperature": 0.
|
| 73 |
},
|
| 74 |
"nl": {
|
| 75 |
-
"ece_after": 0.
|
| 76 |
-
"ece_before": 0.
|
| 77 |
"n": 830,
|
| 78 |
-
"temperature": 0.
|
| 79 |
},
|
| 80 |
"pt": {
|
| 81 |
-
"ece_after": 0.
|
| 82 |
-
"ece_before": 0.
|
| 83 |
"n": 840,
|
| 84 |
-
"temperature": 0.
|
| 85 |
}
|
| 86 |
},
|
| 87 |
-
"ece_after": 0.
|
| 88 |
-
"ece_before": 0.
|
| 89 |
"n": 5000,
|
| 90 |
-
"temperature": 0.
|
| 91 |
},
|
| 92 |
"score": {
|
| 93 |
"by_language": {
|
| 94 |
"de": {
|
| 95 |
-
"ece_after": 0.
|
| 96 |
-
"ece_before": 0.
|
| 97 |
"n": 830,
|
| 98 |
-
"temperature": 0.
|
| 99 |
},
|
| 100 |
"es": {
|
| 101 |
-
"ece_after": 0.
|
| 102 |
-
"ece_before": 0.
|
| 103 |
"n": 840,
|
| 104 |
"temperature": 0.25
|
| 105 |
},
|
| 106 |
"fr": {
|
| 107 |
-
"ece_after": 0.
|
| 108 |
-
"ece_before": 0.
|
| 109 |
"n": 830,
|
| 110 |
"temperature": 0.25
|
| 111 |
},
|
| 112 |
"it": {
|
| 113 |
-
"ece_after": 0.
|
| 114 |
-
"ece_before": 0.
|
| 115 |
"n": 830,
|
| 116 |
-
"temperature": 0.
|
| 117 |
},
|
| 118 |
"nl": {
|
| 119 |
-
"ece_after": 0.
|
| 120 |
-
"ece_before": 0.
|
| 121 |
"n": 830,
|
| 122 |
-
"temperature": 0.
|
| 123 |
},
|
| 124 |
"pt": {
|
| 125 |
-
"ece_after": 0.
|
| 126 |
-
"ece_before": 0.
|
| 127 |
"n": 840,
|
| 128 |
-
"temperature": 0.
|
| 129 |
}
|
| 130 |
},
|
| 131 |
-
"ece_after": 0.
|
| 132 |
-
"ece_before": 0.
|
| 133 |
"n": 5000,
|
| 134 |
-
"temperature": 0.
|
| 135 |
}
|
| 136 |
},
|
| 137 |
"warnings": []
|
|
|
|
| 4 |
"choice": {
|
| 5 |
"by_language": {
|
| 6 |
"de": {
|
| 7 |
+
"ece_after": 0.013708,
|
| 8 |
+
"ece_before": 0.043495,
|
| 9 |
"n": 830,
|
| 10 |
+
"temperature": 6.0
|
| 11 |
},
|
| 12 |
"es": {
|
| 13 |
+
"ece_after": 0.0,
|
| 14 |
+
"ece_before": 5e-06,
|
| 15 |
"n": 840,
|
| 16 |
+
"temperature": 0.05
|
| 17 |
},
|
| 18 |
"fr": {
|
| 19 |
+
"ece_after": 0.002616,
|
| 20 |
+
"ece_before": 0.0096,
|
| 21 |
"n": 830,
|
| 22 |
+
"temperature": 5.0
|
| 23 |
},
|
| 24 |
"it": {
|
| 25 |
+
"ece_after": 0.00251,
|
| 26 |
+
"ece_before": 0.00355,
|
| 27 |
"n": 830,
|
| 28 |
+
"temperature": 3.0
|
| 29 |
},
|
| 30 |
"nl": {
|
| 31 |
+
"ece_after": 0.00271,
|
| 32 |
+
"ece_before": 0.005693,
|
| 33 |
"n": 830,
|
| 34 |
+
"temperature": 4.0
|
| 35 |
},
|
| 36 |
"pt": {
|
| 37 |
+
"ece_after": 0.002972,
|
| 38 |
+
"ece_before": 0.006214,
|
| 39 |
"n": 840,
|
| 40 |
+
"temperature": 5.0
|
| 41 |
}
|
| 42 |
},
|
| 43 |
+
"ece_after": 0.00348,
|
| 44 |
+
"ece_before": 0.011391,
|
| 45 |
"n": 5000,
|
| 46 |
+
"temperature": 5.0
|
| 47 |
},
|
| 48 |
"noul": {
|
| 49 |
"by_language": {
|
| 50 |
"de": {
|
| 51 |
+
"ece_after": 0.001729,
|
| 52 |
+
"ece_before": 0.301935,
|
| 53 |
"n": 830,
|
| 54 |
+
"temperature": 0.1
|
| 55 |
},
|
| 56 |
"es": {
|
| 57 |
+
"ece_after": 8.7e-05,
|
| 58 |
+
"ece_before": 0.303815,
|
| 59 |
"n": 840,
|
| 60 |
+
"temperature": 0.05
|
| 61 |
},
|
| 62 |
"fr": {
|
| 63 |
+
"ece_after": 0.002239,
|
| 64 |
+
"ece_before": 0.299088,
|
| 65 |
"n": 830,
|
| 66 |
+
"temperature": 0.1
|
| 67 |
},
|
| 68 |
"it": {
|
| 69 |
+
"ece_after": 0.005682,
|
| 70 |
+
"ece_before": 0.302041,
|
| 71 |
"n": 830,
|
| 72 |
+
"temperature": 0.1
|
| 73 |
},
|
| 74 |
"nl": {
|
| 75 |
+
"ece_after": 0.006508,
|
| 76 |
+
"ece_before": 0.292156,
|
| 77 |
"n": 830,
|
| 78 |
+
"temperature": 0.15
|
| 79 |
},
|
| 80 |
"pt": {
|
| 81 |
+
"ece_after": 0.000725,
|
| 82 |
+
"ece_before": 0.3015,
|
| 83 |
"n": 840,
|
| 84 |
+
"temperature": 0.1
|
| 85 |
}
|
| 86 |
},
|
| 87 |
+
"ece_after": 0.00323,
|
| 88 |
+
"ece_before": 0.298892,
|
| 89 |
"n": 5000,
|
| 90 |
+
"temperature": 0.15
|
| 91 |
},
|
| 92 |
"score": {
|
| 93 |
"by_language": {
|
| 94 |
"de": {
|
| 95 |
+
"ece_after": 0.001757,
|
| 96 |
+
"ece_before": 0.469981,
|
| 97 |
"n": 830,
|
| 98 |
+
"temperature": 0.15
|
| 99 |
},
|
| 100 |
"es": {
|
| 101 |
+
"ece_after": 0.002654,
|
| 102 |
+
"ece_before": 0.444646,
|
| 103 |
"n": 840,
|
| 104 |
"temperature": 0.25
|
| 105 |
},
|
| 106 |
"fr": {
|
| 107 |
+
"ece_after": 0.002539,
|
| 108 |
+
"ece_before": 0.44168,
|
| 109 |
"n": 830,
|
| 110 |
"temperature": 0.25
|
| 111 |
},
|
| 112 |
"it": {
|
| 113 |
+
"ece_after": 0.004361,
|
| 114 |
+
"ece_before": 0.466521,
|
| 115 |
"n": 830,
|
| 116 |
+
"temperature": 0.15
|
| 117 |
},
|
| 118 |
"nl": {
|
| 119 |
+
"ece_after": 0.003243,
|
| 120 |
+
"ece_before": 0.466974,
|
| 121 |
"n": 830,
|
| 122 |
+
"temperature": 0.15
|
| 123 |
},
|
| 124 |
"pt": {
|
| 125 |
+
"ece_after": 0.009933,
|
| 126 |
+
"ece_before": 0.457518,
|
| 127 |
"n": 840,
|
| 128 |
+
"temperature": 0.15
|
| 129 |
}
|
| 130 |
},
|
| 131 |
+
"ece_after": 0.0108,
|
| 132 |
+
"ece_before": 0.457823,
|
| 133 |
"n": 5000,
|
| 134 |
+
"temperature": 0.15
|
| 135 |
}
|
| 136 |
},
|
| 137 |
"warnings": []
|