structured-cot-ner/qwen3-4b-thinking-structured-noedge-inv-cot-lora

Qwen3-4B-Thinking-2507 · Structured Reasoning without EDGE, MENTIONS before TYPES for NER

Part of the code, data and models of the bachelor thesis Structuring Universal Chain-of-thought for Named Entity Recognition (Department of Computer Science and Engineering, University of Bologna, 2026). All the repositories cited by name belong to the same organization.

Second phase of the project: a teacher reasons twice on the same queries, freely and in a structured format, and students that differ only in the format of their reasoning are trained on the traces. This is a variant of the structured student with a reduced structure.

It is reported in the ablation study of the thesis, among the variants of the structured format.

Given a text and a list of entity types (some present, some absent), the model reasons inside its <think> block with two one-line fields in inverted order: MENTIONS (mentions to extract and candidates rejected), then TYPES (which requested types occur); the EDGE field of the structured format is removed, then answers with a JSON object that maps every requested type to the mentions copied verbatim from the text (an empty list for an absent type).

Training

• Base model: Qwen/Qwen3-4B-Thinking-2507.
• Data: the structured_cot traces of pilener-teacher-cot-agreement-v1 (train, 3,822 queries) with the EDGE line removed and the TYPES and MENTIONS lines swapped, prompt edited accordingly (built with 05_openvocab_sft/make_noedge_data.py --invert); the traces are derived from the existing ones, not regenerated by the teacher. The free and structured traces of a query lead to the same answer, so all the students see the same queries and the same targets.
• Method: LoRA r 16, alpha 32, dropout 0.05 on the q, k, v, o, gate, up and down projections; lr 1e-4, cosine, warmup 3%; 2 epochs; batch 1 x 8 gradient accumulation; bf16; maximum sequence length 5,120; loss on the assistant turn only; 5% validation, best checkpoint by validation loss; seed 42.

Evaluation

Metric: micro F1 with exact, case-insensitive mention matching per type, over the requested types; reasoning length in tokens (generated before the end of the thinking block), mean per query. Queries: every gold type plus 1-3 distractors of the same domain, fixed seed (free-vs-structured-cot-eval). vLLM, T 0.6, top-p 0.95, top-k 20, seed 0, up to 12,288 new tokens. Free and Structured (seed 42) = qwen3-4b-thinking-free-cot-lora and qwen3-4b-thinking-structured-cot-lora, for comparison.

CrossNER (full test sets)

Set	This model F1	Free (seed 42) F1	Structured (seed 42) F1	This model tokens	Free (seed 42) tokens	Structured (seed 42) tokens
AI	0.668	0.686	0.652	231	469	325
Literature	0.716	0.733	0.700	271	563	415
Music	0.777	0.789	0.764	262	583	392
Politics	0.703	0.714	0.694	324	650	436
Science	0.743	0.765	0.749	253	536	363
Mean	0.721	0.737	0.712	268	560	386

Pile-NER test split (in-domain, 2,000 queries)

Set	This model F1	Free (seed 42) F1	Structured (seed 42) F1	This model tokens	Free (seed 42) tokens	Structured (seed 42) tokens
Pile-NER	0.568	0.573	0.562	439	1,029	564

Biomedical benchmarks (500 sentences each, 465 for JNLPBA-Rare)

Set	This model F1	Free (seed 42) F1	Structured (seed 42) F1	This model tokens	Free (seed 42) tokens	Structured (seed 42) tokens
NCBI	0.649	0.699	0.620	183	233	229
BC5CDR	0.731	0.765	0.738	182	311	246
BC2GM	0.491	0.521	0.462	193	257	226
BC4CHEMD	0.567	0.582	0.607	239	291	307
AnatEM	0.331	0.382	0.310	254	292	267
JNLPBA	0.564	0.569	0.563	229	397	280
JNLPBA-Rare	0.527	0.463	0.509	162	336	238
MedMentions-Rare	0.579	0.567	0.585	230	426	291
BioRED	0.654	0.706	0.674	280	522	328
Mean	0.566	0.584	0.563	217	341	268

All predictions and metrics: eval-results (free-vs-structured-cot/ and free-vs-structured-cot-bio/).