member-epflight commited on
Commit
dd86892
·
verified ·
1 Parent(s): 4409c94

Model card: ICLR results, training recipe and figure

Browse files
Files changed (3) hide show
  1. .gitattributes +1 -0
  2. README.md +18 -18
  3. assets/healthbench_size.png +3 -0
.gitattributes CHANGED
@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  tokenizer.json filter=lfs diff=lfs merge=lfs -text
 
 
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  tokenizer.json filter=lfs diff=lfs merge=lfs -text
37
+ assets/healthbench_size.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -40,30 +40,30 @@ datasets:
40
  </p>
41
 
42
  > [!Note]
43
- > We're introducing Apertus-8B-MeditronFO, our small fully open medical specialist LLM, medical specialization of [Apertus-8B-Instruct](https://huggingface.co/swiss-ai/Apertus-70B-Instruct-2509) on the [Fully Open Meditron Corpus](https://huggingface.co/datasets/EPFLiGHT/fully-open-meditron). This model is part of the **Fully Open Meditron** family — the first end-to-end auditable pipeline for clinical LLMs, with open weights, open data, open training recipe, and clinician-vetted corpus construction.
44
 
45
 
46
  - **Part of the Fully Open Meditron family:** End to end fully open clinical LLMs
47
- - **Establishes a new state of the art:** Best medical accuracy on medical benchmarks among small fully open medical LLMs
48
- - **Is prefered over Apertus-70B in 86.9% of comparisons:** Comparison on the clinician validated LM judge evaluation AutoMOOVE.
49
- - **Apertus-8B-MeditronFO improves +13.35 points over its base on aggregate medical benchmarks:** the largest gain in the MeditronFO family.
50
 
51
  <div align="center">
52
- <img src=https://raw.githubusercontent.com/EPFLiGHT/MeditronFO/3caab22b7efaf9e56e2894bac65dab6d3a31aaa0/static/images/plot_fig1_healthbench_timeline_easy_color.png width="100%"/>
53
  </div>
54
 
55
  ## Benchmark
56
 
57
- Accuracy (%) on standard medical benchmarks. See the paper for full evaluation details, confidence intervals, and open-ended Auto-MOOVE results.
58
 
59
  | Benchmark | Apertus-8B-Instruct | **Apertus-8B-MeditronFO** | Δ |
60
  |---|---:|---:|---:|
61
- | MedMCQA | 45.80 | **48.74** | +2.94 |
62
- | MedQA | 51.14 | **58.44** | +7.30 |
63
- | PubMedQA | 37.60 | **75.60** | +38.00 |
64
- | MedXpertQA | 11.71 | **13.67** | +1.96 |
65
- | HealthBench Hard | 21.55 | **38.11** | +16.56 |
66
- | **Average** | 33.56 | **46.91** | +13.35 |
67
 
68
  ## Usage
69
 
@@ -97,15 +97,15 @@ print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_
97
  ## Training
98
 
99
  - **Base model:** [Apertus-8B-Instruct](https://huggingface.co/swiss-ai/Apertus-8B-Instruct-2509)
100
- - **Corpus:** [Fully Open Meditron](https://huggingface.co/datasets/EPFLiGHT/fully-open-meditron) 601k examples (~150M tokens), aggregating eight public medical QA datasets with three clinician-vetted synthetic components: exam-style QA, guideline-grounded QA from 46,469 clinical practice guidelines, and open-ended clinical vignettes
101
- - **Hardware:** 8 NVIDIA GH200 nodes
102
- - **Framework:** Axolotl with FSDP v2 / DeepSpeed ZeRO-3, Flash Attention 2, bf16 mixed precision
103
- - **Decontamination:** System-wide two-stage n-gram and token-alignment decontamination against all evaluation benchmarks
104
 
105
- Full hyperparameters are in Appendix I of the paper.
106
 
107
  ## Compute & footprint
108
- The training was done on 8 nodes of 4 NVIDIA GH200 GPUs for approximately 6 hours on the CSCS Swiss National Supercomputing Centre.
109
  Our trainings have a carbon neutral footprint as the CSCS data center is carbon neutral ([CSCS energy efficiency](https://ethz.ch/en/news-and-events/eth-news/news/2022/11/at-cscs-energy-efficiency-is-a-key-priority-even-at-high-performance.html)).
110
 
111
  ## Limitations & intended use
 
40
  </p>
41
 
42
  > [!Note]
43
+ > We're introducing Apertus-8B-MeditronFO, our small fully open medical specialist LLM, medical specialization of [Apertus-8B-Instruct](https://huggingface.co/swiss-ai/Apertus-8B-Instruct-2509) on the [Fully Open Meditron Corpus](https://huggingface.co/datasets/EPFLiGHT/fully-open-meditron). This model is part of the **Fully Open Meditron** family — the first end-to-end auditable pipeline for clinical LLMs, with open weights, open data, open training recipe, and clinician-vetted corpus construction.
44
 
45
 
46
  - **Part of the Fully Open Meditron family:** End to end fully open clinical LLMs
47
+ - **Best medical accuracy among small fully open medical LLMs:** 48.26% on the five-benchmark average, against 42.39% for Apertus-8B-Instruct
48
+ - **Preferred over Apertus-8B-Instruct in 74.2% of comparisons** on AutoMOOVE, the clinician-validated LLM-judge evaluation (three-judge majority, 516 clinician-written vignettes)
49
+ - **Largest HealthBench gain in the MeditronFO family:** +16.1 points over its base (26.0 → 42.1)
50
 
51
  <div align="center">
52
+ <img src=https://huggingface.co/EPFLiGHT/Apertus-8B-MeditronFO/resolve/main/assets/healthbench_size.png width="100%"/>
53
  </div>
54
 
55
  ## Benchmark
56
 
57
+ Accuracy (%) on four multiple-choice medical benchmarks (greedy decoding) and score (%) on HealthBench (all 5,000 conversations, Gemma-4-31B grader). See the paper for the full evaluation details, confidence intervals and the AutoMOOVE results.
58
 
59
  | Benchmark | Apertus-8B-Instruct | **Apertus-8B-MeditronFO** | Δ |
60
  |---|---:|---:|---:|
61
+ | MedMCQA | 47.8 | **52.0** | +4.2 |
62
+ | MedQA | 52.1 | **60.7** | +8.6 |
63
+ | PubMedQA | 74.3 | **71.4** | −2.9 |
64
+ | MedXpertQA | 11.8 | **15.1** | +3.3 |
65
+ | HealthBench | 26.0 | **42.1** | +16.1 |
66
+ | **Average (5)** | 42.39 | **48.26** | +5.87 |
67
 
68
  ## Usage
69
 
 
97
  ## Training
98
 
99
  - **Base model:** [Apertus-8B-Instruct](https://huggingface.co/swiss-ai/Apertus-8B-Instruct-2509)
100
+ - **Corpus:** [Fully Open Meditron](https://huggingface.co/datasets/EPFLiGHT/fully-open-meditron), 533,888 examples: Curated QA (195.5k, seven public medical QA datasets), Synthetic Curated QA (184.5k), Guidelines QA (129.8k, generated from 14,192 clinical practice guidelines) and Synthetic vignettes (24.0k, modelled on clinician-written MOOVE vignettes). Every answer is written by gpt-oss-120b with rejection sampling (pass@8; multiple-choice answers must match the gold label), with generation prompts co-authored by four clinicians
101
+ - **Decontamination:** two-stage n-gram decontamination against all evaluation benchmarks and the AutoMOOVE test set (4,197 rows removed)
102
+ - **Recipe:** 1 epoch, learning rate 1e-5 with cosine decay and 10% warmup, effective batch 64 sequences of 8,192 packed tokens, AdamW, seed 42
103
+ - **Framework:** Axolotl with DeepSpeed ZeRO-3, bf16
104
 
105
+ Full hyperparameters are in the training appendix of the paper.
106
 
107
  ## Compute & footprint
108
+ The training was done on 2 nodes of 4 NVIDIA GH200 GPUs for 51 min (7 GPU-hours) on the Alps supercomputer of the CSCS Swiss National Supercomputing Centre.
109
  Our trainings have a carbon neutral footprint as the CSCS data center is carbon neutral ([CSCS energy efficiency](https://ethz.ch/en/news-and-events/eth-news/news/2022/11/at-cscs-energy-efficiency-is-a-key-priority-even-at-high-performance.html)).
110
 
111
  ## Limitations & intended use
assets/healthbench_size.png ADDED

Git LFS Details

  • SHA256: b7c6ccaed91aab3d3e93101961f8f29d8c720928a25807b9a4f44bdfcd890bb6
  • Pointer size: 131 Bytes
  • Size of remote file: 121 kB