mimic-vit-biogpt: a documented negative result

A ViT-encoder + BioGPT-decoder checkpoint for chest X-ray impressions whose decoder very likely ignores the image. Kept public as a reproducible record of a silent failure mode, not as a usable model.

Do not use this checkpoint for any image-conditioned task. It has not been evaluated and is not clinically validated.

TL;DR

• Built with VisionEncoderDecoder.from_encoder_decoder_pretrained(ViT, BioGPT) and add_cross_attention=True.
• The BioGPT implementation in Transformers has no cross-attention layers, and the flag is silently ignored (no error or warning). Image features are passed to the decoder but not used.
• The final model is therefore expected to behave like a text-only BioGPT fine-tuned for 500 steps on report impressions.
• A corrected pipeline with a held-out evaluation and a shuffled-image control is planned. Nothing here is a benchmark result.

Model details

Developer	Vikhram S
Intended design	VisionEncoderDecoderModel: ViT-Base/16 (google/vit-base-patch16-224-in21k) + BioGPT (microsoft/BioGPT) with a linear projection (768 to 1024)
Size	About 434M parameters (computed from the printed architecture; checkpoint about 1.74 GB, fp32)
Task as intended	Chest X-ray to impression text
License	Apache-2.0 for these weights; see the base-model pages for their terms
Status	Superseded; known issue

What went wrong, and the evidence

Evidence	Observation
Decoder layers	The printed model shows only self_attn and feed-forward blocks in each of the 24 BioGPT layers, with no cross-attention module.
Library source	In Transformers 5.18 the BioGPT modeling file defines no encoder-attention module and has no encoder_hidden_states argument. VisionEncoderDecoderModel passes that keyword, and BioGPT's **kwargs absorbs it. Check other versions yourself.
Checkpoint size	A model with no cross-attention has about 433.9M parameters (1.74 GB in fp32), matching the file on the Hub. With 24 added cross-attention blocks it would be about 534.8M (2.14 GB).
Training loss	About 1.86 at step 0 and 1.8 to 2.0 at step 480 (average 1.99), with no downward trend over 500 steps. This is consistent with, but not proof of, a decoder that cannot use the image.

The loss curve is weak evidence on its own, since a short run with a small learning rate could look similar for other reasons. The structural evidence is the strong part.

Reproduce the check:

import torch
from transformers import VisionEncoderDecoderModel

m = VisionEncoderDecoderModel.from_pretrained("Vikhram-S/mimic-vit-biogpt")

# 1) Does the decoder define any cross-attention parameters?
print([n for n, _ in m.decoder.named_parameters()
       if "cross" in n.lower() or "encoder_attn" in n.lower()])

# 2) Does the loss reach the image encoder?
m.train()
out = m(pixel_values=torch.randn(1, 3, 224, 224),
        labels=torch.randint(5, 1000, (1, 16)))
out.loss.backward()
n_grad = sum(p.grad is not None and p.grad.abs().sum() > 0
             for p in m.encoder.parameters())
print("encoder tensors with nonzero gradient:", int(n_grad),
      "of", len(list(m.encoder.parameters())))

# 3) Do different images give identical greedy output?
m.eval()
with torch.no_grad():
    a = m.generate(pixel_values=torch.randn(1, 3, 224, 224),
                   max_length=30, num_beams=1, do_sample=False)
    b = m.generate(pixel_values=torch.randn(1, 3, 224, 224),
                   max_length=30, num_beams=1, do_sample=False)
print("identical:", a.shape == b.shape and bool((a == b).all()))

Results of this script on the released weights will be added in a card revision.

Training details

Item	Value
Data	itsanmolgupta/mimic-cxr-dataset-cleaned, train split (16,000 examples)
Target text	impression field only, truncated to 128 tokens
Objective	Token-level cross-entropy
Optimizer / schedule	AdamW, lr 3e-5, 10% linear warmup then linear decay
Precision	Mixed precision (AMP)
Batch size / steps	4 / 500 (about 2,000 examples, one partial epoch)
Hardware	One NVIDIA T4 (Google Colab)
Training code	train_colab.ipynb in this repository
Compute	Not measured; 500 optimizer steps on one T4

The validation and test splits were not used, and no validation loss was computed.

Evaluation

None. No language-generation or clinical-label metrics were computed for this checkpoint. The 2,000-example test split of the source dataset is unused and is reserved for the corrected model.

Uses

• Appropriate: studying this failure mode, or as a minimal example of a silent cross-attention omission.
• Inappropriate: report generation, clinical or diagnostic use, or any claim about image-conditioned performance.

Not implemented

No explainability or attribution method, no confidence estimation, no retrieval grounding, no structured findings/impression output (the target was impressions only), and no alignment or consistency losses.

Bias, risks and limitations

• The decoder likely ignores the image, so outputs may be unrelated to the input.
• Very short training on a single-institution dataset processed by a third party; demographic and preprocessing bias are uncharacterized.
• No error analysis or hallucination analysis was performed.

Data and access terms

This repository contains weights only. MIMIC-CXR is distributed through PhysioNet under a credentialed-access agreement; users are responsible for complying with its terms, including when using processed copies.

Recommendations for people building similar models

1. Before training, assert that cross-attention parameters exist in the decoder.
2. After a few steps, check that the encoder receives nonzero gradients.
3. Always evaluate with a shuffled-image control; a model that scores the same with the wrong images is not using images.

Plan

A corrected model (a decoder with cross-attention, or an image-prefix design) trained and evaluated on the held-out test split, with the shuffled-image control, will be released as a separate repository. Results will be reported only with the script that produced them.

Citation

@misc{vikhram2026mimicvitbiogpt,
  author = {Vikhram S},
  title  = {mimic-vit-biogpt: a documented negative result},
  year   = {2026},
  howpublished = {Hugging Face model repository},
  url    = {https://huggingface.co/Vikhram-S/mimic-vit-biogpt}
}

Author

Vikhram S: Google Scholar | GitHub