FunDLM-Medium-1024bp-H3K4me3

LoRA adapter that classifies 1,024-bp DNA windows as carrying a ChIP-seq peak of H3K4me3 (1) or not (0). It was fine-tuned on top of the fungal DNA language model FunDLM-Medium-1024bp and released with the FunDLM paper.

H3K4me3 is associated with actively transcribed genes.

Model summary

Base architecture	FunDLM-Medium-1024bp (Caduceus; bidirectional Mamba-2 with reverse-complement parameter sharing; 64 layers, d_model 768, 471M unique parameters)
Max input length	1,024 bp
Task	Sequence classification (softmax over two classes)
Number of classes	2
Labels	0 = non-peak window, 1 = H3K4me3 peak window
Adapter	LoRA, r = 8, alpha = 32, dropout = 0.1
Training checkpoint	checkpoint-3621 (3,621 optimization steps, 128 windows per step)

Repository contents

Path	Contents
adapter/adapter_config.json, adapter/adapter_model.safetensors	LoRA adapter weights
config.json, configuration_caduceus.py, modeling_caduceus.py, modeling_rcps.py	base FunDLM-Medium-1024bp configuration and custom model code
pytorch_model.bin	base FunDLM-Medium-1024bp weights
tokenizer.json, tokenizer_config.json, special_tokens_map.json	base tokenizer

The base model files are included so the repository is self-contained. The same base model is also available as gnikesh/FunDLM-Medium-1024bp.

Installation

Environment setup follows the same steps as PlantCAD2. This model uses a Mamba-based state-space backbone, so the Mamba kernels are required:

# Core dependencies (versions used for fine-tuning)
pip install transformers==4.49.0 peft==0.14.0 datasets==3.3.2
pip install torch  # choose the build that matches your CUDA version

# Mamba/Caduceus-specific dependencies
pip install mamba_ssm causal_conv1d --no-build-isolation

Running the classifier in Python

The example assumes a CUDA GPU (the base checkpoint is stored in bfloat16). On CPU, cast the model with model.float() and run the forward pass without the autocast block.

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from peft import PeftModel

base_id = "gnikesh/FunDLM-Medium-1024bp"
adapter_id = "PlantG3/FunDLM-Medium-1024bp-H3K4me3"

tokenizer = AutoTokenizer.from_pretrained(base_id, trust_remote_code=True)
base = AutoModelForSequenceClassification.from_pretrained(
    base_id,
    trust_remote_code=True,
    num_labels=2,
    id2label={0: "non-peak", 1: "peak"},
    label2id={"non-peak": 0, "peak": 1},
)
# Use the base classifier with the adapter applied, without merging it.
model = PeftModel.from_pretrained(base, adapter_id, subfolder="adapter").base_model.model
model = model.cuda().eval()

sequence = "ACGT" * 256  # replace with a 1,024-bp window
encoded = tokenizer(
    sequence,
    add_special_tokens=False,
    padding="max_length",
    truncation=True,
    max_length=1024,
)
input_ids = torch.tensor(encoded["input_ids"]).unsqueeze(0).cuda()
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
    logits = model(input_ids=input_ids).logits
prob_peak = torch.softmax(logits.float(), dim=-1)[0, 1].item()
print(f"H3K4me3 peak probability: {prob_peak:.4f}")

Do not call merge_and_unload() on this adapter. The forward and reverse-complement Mamba branches carry separate LoRA weights, and merging folds both into the shared base weights and changes the predictions.

Batch prediction with the FunDLM code

The fine-tuning and prediction CLI is in the FunDLM repository. Clone this model repository and point --checkpoint_dir at its adapter/ directory:

git clone https://huggingface.co/PlantG3/FunDLM-Medium-1024bp-H3K4me3

python src/finetune/lora_fine_tune.py predict \
    --checkpoint_dir FunDLM-Medium-1024bp-H3K4me3/adapter \
    --data_dir windows.parquet \
    --output_file predictions.csv \
    --model_name gnikesh/FunDLM-Medium-1024bp \
    --task_type classification \
    --batch_size 32 \
    --bf16 True \
    --seed 42

windows.parquet is the tokenized input produced by the tokenize subcommand of the same script. Inputs should be exactly 1,024 bp; the tokenizer pads shorter sequences and maps non-ACGT characters to [UNK].

Training data

The adapter was trained on 1,024-bp windows from the Br48, Guy11 and O135 isolates. Peak intervals of H3K4me3 were extended by 100 bp on either side and tiled with a 182 bp step, and control windows came from the non-overlapping mappable complement of the peaks. Windows containing non-ACGT bases or conflicting labels were removed, each sequence was kept once, and test windows sharing a sequence with a training isolate were discarded. O137 was held out entirely for testing. O219 has no H3K4me3 data, so this mark is trained on the three isolates that were assayed.

This gave 154,594 training, 8,137 validation and 54,258 test windows, with positive fractions of 0.430, 0.433 and 0.445.

Training settings

• LoRA rank 8, alpha 32, dropout 0.1, applied to the Mamba in_proj and out_proj projections
• AdamW, three epochs at 128 windows per step (16 per device x 8 gradient-accumulation steps)
• Learning rate 1e-4 with cosine decay and 100 warm-up steps
• Weight decay 0.01, gradient clipping 1.0, bfloat16, seed 42
• Reported checkpoint: checkpoint-3621

Evaluation

Held-out test set: 54,258 O137 windows (positive fraction 0.445).

Metric	Value
AUROC	0.996
Average precision (AP)	0.995
Accuracy at probability 0.5	0.983
Precision at probability 0.5	0.982
Recall at probability 0.5	0.981
F1 at probability 0.5	0.981

Genome-wide O137 scan (43,159 non-overlapping 1,024-bp windows, prevalence 15.3%): AUROC 0.980, AP 0.910.

Limitations

The O137 test windows were sampled around peaks, so the positive fraction is much higher than in a genome and the genome-wide AP is lower than the test-window AP. A probability threshold of 0.5 overestimates the extent of the mark genome-wide; use the probabilities to rank windows and calibrate a threshold on a related genome with ChIP-seq data. The isolate-held-out test mostly measures transfer between closely related genomes; when whole chromosomes were withheld from all isolates, the pooled AUROC was 0.964. A logistic regression on 6-mer frequencies reached test-window AP 0.839, well below this adapter's 0.995, so the adapter captures more than simple sequence composition.

Contact

PlantG3 lab, Kansas State University. Please open an issue at PlantG3/funDLM for questions about the model.