Qwen-jeff-B2

A pointwise decision scorer. You give it one instruction, one state and a declared set of options. It gives you a probability distribution over exactly those options.

Inspired by Typesafe's Jev model architecture.

z_i = f(state, instructions, option_i)        p = softmax(z) over the declared options

Each option is scored on its own forward pass with the full context; the softmax runs over the candidate set. The answer carries probabilities, argmax, max_probability and entropy.

backbone	Qwen/Qwen3-1.7B, causal
readout	hidden state at the last real token of \n### Score: → LayerNorm(2048) → Linear(2048, 1)
parameters	1,720,581,121 (backbone 1,720,574,976 + head 6,145)
precision	BF16
context	2,048 tokens per candidate
trained on	102,391 decisions, 2 epochs, 1× H200, seed 0
fitted temperature	1.1620 (calibration split only)
automation threshold	0.9588 (≤5% error on the locked test, 19.13% coverage)

model/backbone/ bundles five tokenizer/config files copied from Qwen/Qwen3-1.7B (config.json, tokenizer.json, tokenizer_config.json, vocab.json, merges.txt, ~15.9 MB total) so the model loads offline. Qwen3-1.7B is licensed Apache-2.0; these files are redistributed unmodified under that licence, separately from the Apache-2.0 licence on this project's own code and weights.

---

The three primitives

primitive	what you declare	what you get
Noul	two options (default yes / no)	a 0–1 probability; the reported number is P(first option)
Choice	2–255 named, described options	a distribution over those options
Score	2–10 ordered levels	a distribution over the levels, plus expected_level and level_variance

---

Query it

Command line

# a Choice
qwen-jeff ask choice \
  --state "Ticket #4471. Customer writes: 'I reset my password twice this morning
           and the app still says invalid credentials. I can log in on the website
           but not in the iOS app. My subscription renewed last week and the charge
           went through fine.'" \
  --instruction "Route this ticket to exactly one queue." \
  --option "billing: payment, invoices, refunds and subscription charges" \
  --option "technical: the product does not work as intended" \
  --option "sales: pricing, upgrades and new contracts" \
  --option "account_closure: the customer wants to cancel"

── choice · 4 candidates ─────────────────────────────────────────────────
   Qwen/Qwen3-1.7B @ step 3200 · mps/bfloat16 · temperature 1.1620

  question  Route this ticket to exactly one queue.
  state     Ticket #4471. Customer writes: 'I reset my password twice this
            morning and the app still says invalid credentials. I can log
            in on the website but not in the iOS app. My subscription
            renewed last week and the charge went through fine.'

  ▸ technical        0.7040  █████████████████████
    account_closure  0.1364  ████
    billing          0.1328  ███▉
    sales            0.0268  ▊

  argmax    technical  at p = 0.7040
  entropy   0.8839 of 1.3863 nats  ███████▋     63.8% of maximum
  automate  no  p < 0.9588 (<=5% error on the locked test, seen families)
  latency   0.189 s
──────────────────────────────────────────────────────────────────────────

# a Noul — a 0-1 probability (a two-option decision under the hood)
qwen-jeff ask noul --state @clause.txt \
  --instruction "Does the agreement permit sublicensing to affiliates?"

# a Score — five levels, or named ones
qwen-jeff ask score --state @answer.txt \
  --instruction "Rate the helpfulness of the response." --levels 5
qwen-jeff ask score --state @bug.txt \
  --instruction "Rate the severity of this defect." \
  --levels "trivial,minor,moderate,major,critical"

# many at once, in the Decision JSON format
qwen-jeff ask choice --batch decisions.jsonl --out answers.jsonl

--state and --instruction take text or @path. Other flags: --json (machine-readable), --no-temperature, --temperature-value, --model, --max-tokens (default 2048), --device {auto,mps,cpu,cuda}, --dtype {auto,bfloat16,float32}, --no-color, --state-chars, --max-padded-tokens, --max-rows-per-forward.

Python

from qwen_jeff import load_scorer, ask

scorer = load_scorer("model")            # or the QWEN_JEFF_MODEL path

answer = ask(
    scorer, "choice",
    state="Ticket #4471. The customer cannot log in after a password reset.",
    instructions="Route this ticket to exactly one queue.",
    options=[
        "billing: payment, invoices and refunds",
        "technical: the product does not work as intended",
        "sales: pricing, upgrades and new contracts",
    ],
)
print(answer.argmax, round(answer.max_probability, 4))   # c1 0.7040
print(answer.probabilities)                              # {'c0': ..., 'c1': ..., 'c2': ...}

# a Noul: the reported number is P(first option)
noul = ask(scorer, "noul",
           state=open("clause.txt").read(),
           instructions="Does the agreement permit sublicensing to affiliates?")
p_yes = noul.probabilities[next(iter(noul.probabilities))]

# a Score: ordered levels, with an expectation over them
score = ask(scorer, "score",
            state="The answer is short but correct.",
            instructions="Rate the helpfulness of the response.",
            levels="5")
print(score.expected_level, score.level_variance)

# raw softmax, without the fitted temperature
raw = ask(scorer, "noul", state="...", instructions="...", temperature=False)

For full control, build the Decision yourself and score it:

from qwen_jeff import Decision, Candidate, Primitive, score_one

decision = Decision(
    decision_id="d1",
    state="Premise: ...\nHypothesis: ...",
    instructions="Does the premise entail the hypothesis?",
    primitive=Primitive.CHOICE,
    candidates=[
        Candidate(candidate_id="c0", name="entailment", description="the premise entails it"),
        Candidate(candidate_id="c1", name="neutral", description="neither entailed nor contradicted"),
        Candidate(candidate_id="c2", name="contradiction", description="the premise contradicts it"),
    ],
)
answer, seconds = score_one(scorer, decision, temperature=scorer.temperature)

Score one decision at a time

score_many loops over score_one; it does not batch, and PredictLimits defaults to max_decisions=1. That is not a performance oversight. See batch-composition sensitivity below.

Latency

MacBook, MPS/BF16, model resident, one decision at a time:

	seconds
weights load (once per process)	16.6 – 19.8
2-option Noul	0.035 – 0.045
3–5-option Choice or 5-level Score	0.06 – 0.13
20-option Choice, ~90-token state	0.34

Load dominates. Keep the model in one process, or use the batch path.

---

Evaluation

Every number in this section comes from a single read of a frozen locked test. The checkpoint was fixed at step 3,200 before the evaluation was submitted; the temperature was fitted on a separate calibration split; every other choice was made on a development split.

Headline

metric	value	95% document bootstrap (1,108 groups, 2,000 resamples)
accuracy	0.7375	[0.7054, 0.7686]
macro-over-family accuracy	0.7375	—
macro F1	0.6165	—
NLL	0.6787	[0.6096, 0.7502]
Brier	0.3739	[0.3367, 0.4125]
ECE, 15-bin equal-count	0.0534	—
ECE, 10-bin equal-width	0.0494	—
Score MAE (expected / argmax, n=400)	0.9275 / 0.8675	—
Score RPS	0.1463	—
schema validity	1,200 / 1,200 answers valid, 0 issues	—

Per family

family	accuracy	macro F1	NLL
entailment	0.8025	0.7983	0.5663
relevance	0.6725	0.4801	0.7911

Coverage at ≤5% error

19.13% coverage at a realised risk of 4.58% — 153 decisions of 800 accepted above a 0.9588 probability threshold. At ≤10% error: 47.63% coverage at 9.97% realised risk, threshold 0.8511.

This is the number the automation threshold in config.json comes from. It is a property of that split and those families.

Temperature and calibration transfer

Fitted on the calibration split only. Fitted temperature 1.1620 (calibration NLL 0.6454 at T=1 → 0.6394 at the fitted T).

split		NLL	Brier	ECE(15)	ECE(10)	accuracy
calibration (where it was fitted)	before	0.6454	0.3606	0.0582	0.0438	0.7288
calibration	after	0.6394	0.3578	0.0342	0.0336	0.7288
locked test	before	0.6787	0.3739	0.0534	0.0494	0.7375
locked test	after	0.6670	0.3707	0.0442	0.0352	0.7375

A temperature above 1 means the model was over-confident, mildly. Scaling cuts calibration-split ECE by 41% and locked-test ECE by 17%, moves NLL and Brier slightly, and leaves accuracy untouched.

The gain transfers only partially. The fit improves the split it was fitted on roughly twice as much as the one it is applied to, and that gap is the honest measure of what calibration transfer is worth here.

Invariance checks

check	n	max abs deviation	tolerance	argmax changes	result
permutation equivariance, FP32 on CPU	35	1.79e-07	atol 1e-6, rtol 1e-5	0	pass
batch invariance (singleton vs batched)	12	0.0	atol 1e-5	0	pass

All 35 equivariance probes are within tolerance and all 5 duplicate-candidate pairs are bitwise equal: reordering your options does not change the answer. Batch invariance is exactly 0.0 here — which is what a fixed singleton shape buys, and must not be read as contradicting the 5.5e-2 across different batch compositions reported under limitations. That evaluation deliberately never varied the composition.

State controls — the model reads the state

variant	accuracy	NLL	Brier
full state	0.7325	0.679	0.374
shuffled state	0.3700	1.689	0.900
removed state	0.4113	1.556	0.879

-36.25pp when the state is shuffled, −32.13pp when it is removed. For contrast, the project's control system (an off-the-shelf scorer on a much smaller training pool) beat its own shuffled-context control by −0.12pp on this same split: its accuracy was what the candidate set alone supported. This model's accuracy is a property of the state it was given.

Against the control

On the identical 800-decision subset both systems cover:

system	accuracy	entailment	relevance	macro F1	NLL	Brier	ECE(15)	coverage @≤5% error
control (4,138-decision pool, CPU)	0.3800	0.3025	0.4575	0.174	1.302	0.727	0.142	0.125% (1 decision)
Qwen-jeff-B2	0.7375	0.8025	0.6725	0.6165	0.679	0.374	0.053	19.13% (153)

---

Limitations

Read these before you put a probability from this model into anything that acts on it.

1. Trained and evaluated on seen families only. The training corpus covers entailment, relevance, rubric response assessment, classification, routing, multiple choice and verifiable.

2. The evaluation measures document generalisation, from near single-source families. The locked test is held-out document groups from the same sources the model trained on, and each evaluated family is effectively single-source: relevance ~100% ESCI, entailment ~96% MNLI.

3. The calibration was fitted on seen families. T=1.1620 comes from the calibration split — entailment, relevance and rubric assessment, on sources the model trained on — and it already transfers only partially between two splits of those same families. Applied to an invoice field, a security alert or anything else outside that mixture it is an extrapolation: the probabilities stay ranked as the model ranked them, but their calibration is unwarranted. Pass --no-temperature (or temperature=False) for the raw softmax. The 0.9588 automation threshold carries the same caveat twice over: it is the ≤5% error point of the locked test at 19.13% coverage.

4. Batch-composition sensitivity — score one decision at a time. On this trained checkpoint, the same decision scored under different batch compositions moved by up to 5.5e-2 in probability (median deviation exactly 0.0; ~10% of decisions exceeded 1e-2). Operationally it barely showed — at most 1 argmax flip in 860 and 0 crossings of the automation threshold — but it is ordinary batched-reduction behaviour and there is no defensible tolerance below 1e-1. Do not compare probabilities from this model across batch compositions. The released code therefore scores one decision per forward pass, which is also the shape every number above was measured in.

6. Single seed. Everything here is seed 0. The run-to-run spread is unmeasured. There are no extra seeds.

7. Long states are truncated The budget is 2,048 tokens per candidate, with the instruction capped at 384 and the candidate at 384; the state is truncated head 60% / tail 40% with an explicit marker: answer.truncated.

8. max_probability and entropy are not correctness probabilities. They are shape statistics of the distribution. Their relationship to correctness is the calibration question answered, partially, above,

9. English only. The corpus is English throughout.

---

Training data

102,391 decisions, built from fourteen wired sources.

Sources, decision counts and licences

Licences are as recorded in the project's source audit (data/audit/tasksource_families.csv, data/audit/additional_sources.csv), taken from each dataset's Hugging Face card at audit time. ⚠ marks a licence that is share-alike, non-commercial, unknown, research-only or otherwise not clearly permissive — see the warning under the table.

source (audit id)	family	primitive(s)	decisions	licence
glue/mnli	entailment	noul, choice	11,712	other (undeclared)
snli	entailment	noul, choice	8,971	CC BY-SA 4.0
super_glue/cb	entailment	noul, choice	219	other (undeclared)
glue/sst2	classification	noul	8,965	other (undeclared)
glue/cola	classification	noul	7,954	other (undeclared)
civil_comments (toxicity)	classification	noul	7,488	CC0-1.0
dbpedia_14	routing	choice (14)	8,978	CC BY-SA 3.0
ag_news	routing	choice (4)	8,972	unknown
lex_glue/ledgar	routing	choice (100)	2,500	CC BY-4.0
openbookqa	multiple choice	choice (4)	4,431	unknown
nvidia/HelpSteer2 (helpfulness)	rubric response assessment	score (5)	8,964	CC BY-4.0
nvidia/HelpSteer2 (correctness)	rubric response assessment	score (5)	8,964	CC BY-4.0
tasksource/esci	relevance	score, choice	5,670	Apache-2.0
wiki_qa	relevance	noul, choice	675	other (undeclared)
synthetic (generated by this project)	verifiable + multiple choice	mixed	7,928	project-generated
total			102,391

Family totals over the whole corpus: classification 24,407 · entailment 20,902 · routing 20,450 · rubric response assessment 17,928 · multiple choice 7,077 · relevance 6,345 · verifiable 5,282. The synthetic slice is the only one spanning two families (it supplies the whole verifiable family and the balance of multiple choice) and carries the instruction-flip pairs.

Primitive mix: choice 0.535 / noul 0.264 / score 0.201. Option-count bands: 2–8 88.45%, 9–32 9.11%, 33–255 2.44%. Duplication factor 6.729 (each decision's context is re-encoded once per candidate, which is what pointwise scoring costs).

---

Licence

Apache-2.0 for the code and the released weights. See LICENSE, and the licence warning above regarding the upstream training data.

Citation

@software{qwen_jeff_b2_2026,
  title  = {Qwen-jeff-B2: an instruction-conditioned pointwise decision scorer},
  year   = {2026},
  url    = {https://huggingface.co/divergentlabs/qwen-jeff-B2}
}