# PlasmidLMEval

Comprehensive evaluation data for PlasmidLM models. Generated by the eval pipeline in [McClain-Thiel/PlasmidLLM](https://github.com/McClain-Thiel/PlasmidLLM/tree/main/eval).

## NeurIPS 2026 rebuttal snapshot

The complete rebuttal artifact snapshot is in
[`neurips_2026_rebuttal/`](neurips_2026_rebuttal/README.md). It contains the
held-out-registry inputs, exact G7e runtime, checkpoints, four-seed generations,
raw annotations, evaluation outputs, source scripts, W&B lineage inventory, and
SHA-256 manifest needed to reproduce and audit the reported results.

Verified snapshot: 546 files, 20,368,130,505 bytes. The machine-readable file
manifest is at
[`neurips_2026_rebuttal/lineage/local_file_manifest.json`](neurips_2026_rebuttal/lineage/local_file_manifest.json)
and the checksum manifest is at
[`neurips_2026_rebuttal/lineage/SHA256SUMS`](neurips_2026_rebuttal/lineage/SHA256SUMS).

## Structure

```
neurips_2026_rebuttal/                  # NeurIPS 2026 rebuttal reproducibility archive
  README.md                             # Canonical experiment, results, and archive guide
  lineage/                              # SHA-256, S3, HF, and W&B provenance records
  source/                               # Evaluation, validation, and orchestration scripts
  s3/neurips_rebuttal/                  # Complete snapshot of the source S3 prefix

reference/                              # Addgene-500 reference panel
  addgene_500.fasta                     # 500 Addgene plasmid sequences
  addgene_reference_500.csv             # ID, name, sequence, length
  addgene_reference_metrics.csv         # Pre-computed: length, GC, longest ORF, MFE density
  addgene_500_3mer_freqs.json           # Canonical 3-mer frequency distribution
  addgene_500_mfe_per_seq.parquet       # Per-sequence MFE density (DNA Mathews 2004,
                                        #   5×2 kb random windows for >5 kb seqs)
  training_sigs.zip                     # Sourmash signature of training set (20644 plasmids,
                                        #   k=31, scaled=100)

config/                                 # Eval configuration files
  environment.yml                       # Pinned tool versions (plannotate 1.2.2,
                                        #   prodigal 2.6.3, etc.)
  feature_categories.yaml               # pLannotate feature → eval category mapping

runs/
  firstlight_20260408/                  # Phase 1: sampling config grid search
    sweep_results.csv
    sweep_summary.json
    t{temp}_p{top_p}_r{rep}/
      generations.{parquet,fasta}
      plannotate_results.json
      cell_summary.json

  grpo_plannotate_full_20260408/        # Headline GRPO eval (n=1000, seed 42)
    config.json
    generations.{parquet,fasta}
    matched_pairs.parquet               # Paired with closest real Addgene plasmid
    plannotate_results.json
    dustmasker/{dustmasker.intervals,input_filtered.fasta}
    orfs/{prodigal.gff,prodigal_proteins.faa}
    metrics/
      tier1_summary.json
      tier1_distributional.parquet      # Per-seq length, GC, ORF density, ...
      tier3_essentials.parquet          # Per-seq has_ori, has_marker, plausibility_pass, ...
      tier5_architecture.parquet
      extra_metrics.json                # Codon JSD, GC skew, MFE summary
      prompt_fidelity.json
      prompt_fidelity_per_seq.parquet   # Per-prompt sseqid-intersection fidelity
      mfe_per_seq.parquet               # NEW (2026-04-27): per-seq MFE density
      memorization.parquet              # Sourmash hits ≥ threshold (typically empty)
      gen_sigs.zip
      shap_summary.png
      diversity.json
      discriminator.json
    evo2/
      evo2_scores.parquet               # Per-pair PPL on n=200 matched real/gen
      evo2_summary.json
      gen_embeddings.npy                # 200×4096 mean-pooled embeddings
      real_embeddings.npy               # 200×4096

  base_kmer6_matched_20260420_1303/     # Pre-GRPO base eval (n=1000, seed 42; same prompts as GRPO)
    config.json
    generations.{parquet,fasta}
    matched_pairs_evo2.parquet          # NEW (2026-04-27): 200-pair subset for Evo2 scoring
    plannotate_results.json
    dustmasker/, orfs/
    metrics/                            # Same schema as GRPO
      mfe_per_seq.parquet               # NEW (2026-04-27)
      tier{1,3,5}_*, extra_metrics.json, prompt_fidelity_per_seq.parquet, ...
    evo2/                               # NEW (2026-04-27)
      evo2_scores.parquet               # 200 pairs, paired with same real plasmids as GRPO Evo2 run
      evo2_summary.json

  bon_{base,grpo}_seed{43,44,45}_20260420_1303/   # Best-of-K seeds (paired with seed-42 above)
    Same schema as base_kmer6_matched / grpo_plannotate_full minus the evo2/ subdir.

  base_kmer6_sweep_20260420_1303/       # Pre-GRPO base temperature sweep (n=100/cell)
    t{03,05,10}/                        # No t07 — seed 42 above already provides t=0.7 at n=1000
      Same schema, no evo2/

  forced_sp_grpo_20260427/              # NEW (2026-04-27): n=400 GRPO generations forced
                                        #   to SP_ECOLI/SP_YEAST prompts only (200/200 with
                                        #   replacement from 40 unique SP_ECOLI + 99 unique
                                        #   SP_YEAST training prompts). Same sampling config
                                        #   as the matched runs.
    generations.{parquet,fasta}
    orfs/prodigal.gff
    metrics/{cai_per_orf,cai_per_seq}.parquet

  forced_sp_base_20260427/              # Same as above for the supervised base.

reference/training_sp_sample/           # NEW (2026-04-27): n=300/SP × 10 SP classes from
                                        # training_pairs_v4. Real plasmids stratified by SP
                                        # token, used as the positive control for the
                                        # CAI-by-SP_token experiment.
    generations.parquet                 # (mis-named for layout convenience — these are real
                                        # training plasmid sequences with their original
                                        # prompt strings; "generations" matches the schema)
    orfs/prodigal.gff
    cai_per_orf.parquet
    cai_per_seq.parquet

figures/                                # NEW (2026-04-27): rendered paper figures, archived
                                        # alongside the data they were rendered from.
    cai_yeast_minus_ecoli_forest.{pdf,png}

eval_log.md                             # Free-form log of every run that produced data here
```

## Models Evaluated

- **`McClain/PlasmidLM-kmer6`** — pre-GRPO supervised base (19.3 M params, dense kmer6 transformer).
  Acts as the supervised promptable baseline.
- **`McClain/PlasmidLM-kmer6-GRPO-plannotate`** — GRPO post-trained from above with pLannotate composite reward.

## Headline numbers (paired n=1000, seed-42 matched)

| Metric | Base | GRPO | Δ |
|---|---:|---:|---:|
| plausibility_pass (`ori AND marker AND (prom OR cds)`) | 56.8 % | 59.6 % | +2.8 pp |
| prompt fidelity (sseqid intersection, mean over 996 prompts) | 35.5 % | 41.6 % | +6.1 pp |
| useful @ T=0.5 (plausibility ∧ fidelity ≥ 0.5), single-shot | 38.7 % | 48.5 % | +9.8 pp |
| useful @ T=0.5, best-of-4 | 78.9 % | 89.7 % | +10.8 pp |
| Evo 2 PPL ratio gen/real (median, n=200) | 1.10 | 1.19 | +0.09 |
| MFE density mean (kcal/mol/nt, n=500, 5×2 kb windows) | -0.146 | -0.158 | -0.012 |
| MFE density (real Addgene-500) | — | — | -0.163 |
| CAI yeast−ecoli, SP_ECOLI prompts (forced n≈100) | +0.003 (n.s.) | **−0.019**\*\* | matches real (−0.024) |
| CAI yeast−ecoli, SP_YEAST prompts (forced n≈265) | **+0.027**\*\* | −0.008 (n.s.) | base matches real (+0.039) |

Note: GRPO improves prompt fidelity and the useful-plasmid rate, but is slightly *less* in-distribution under Evo 2 than base. MFE moves in the opposite direction — GRPO is more thermodynamically stable and closer to real Addgene than base. Evo 2 PPL captures global "naturalness" under a 7 B-parameter generic DNA LM; MFE captures local structure stability of folded RNA. The two move in opposite directions because they measure different things.

## Host-specific codon usage (SP × CAI, 2026-04-27)

To test whether the model conditions on the soft `SP_*` token in a way pLannotate cannot
detect, we score per-ORF Codon Adaptation Index (Sharp & Li, NAR 1987) against
three host codon-usage tables (E. coli K-12 / S. cerevisiae / human; Kazusa
via `python_codon_tables`) on:

  1. **Real plasmids** stratified by their SP token (training subsample, 300/class).
     Establishes that real plasmids carry a host-specific codon signal.
  2. **Forced SP_ECOLI / SP_YEAST generations** from base and post-GRPO
     (n=400 each, with-replacement sampling from the 40 unique SP_ECOLI and 99
     unique SP_YEAST prompts in training).
  3. The original n=1000 paired eval (only SP_HUMAN/SP_MOUSE have meaningful n
     in that set; SP_ECOLI/SP_YEAST appear in only ~10 prompts each).

Per-sequence Δ = $\overline{\textrm{CAI}}_\textrm{yeast} - \overline{\textrm{CAI}}_\textrm{ecoli}$,
length-weighted across ORFs ≥ 100 codons in each sequence. Bootstrap 95% CIs on
the mean of Δ within each (source, SP) cell.

| source | SP token | n | Δ ± 95% CI | direction matches real? |
|---|---|---:|---:|---:|
| **real (training, positive control)** | SP_ECOLI | 300 | −0.024 [−0.035, −0.014] | — |
| **real** | SP_YEAST | 300 | +0.039 [+0.029, +0.049] | — |
| **real** | SP_HUMAN | 300 | −0.006 [−0.012, −0.001] | — |
| **real** | SP_MOUSE | 300 | −0.013 [−0.019, −0.007] | — |
| forced base | SP_ECOLI | 98 | +0.003 [−0.018, +0.024] | **no** (n.s.) |
| forced base | SP_YEAST | 263 | +0.027 [+0.016, +0.038] | yes |
| forced GRPO | SP_ECOLI | 107 | −0.019 [−0.036, −0.002] | yes |
| forced GRPO | SP_YEAST | 266 | −0.008 [−0.018, +0.002] | **no** (n.s. / sign flip) |
| eval-set base | SP_HUMAN | 117 | −0.021 [−0.038, −0.005] | yes (overshoots) |
| eval-set base | SP_MOUSE | 41 | +0.016 [−0.009, +0.041] | **no** (sign flip) |
| eval-set GRPO | SP_HUMAN | 114 | −0.004 [−0.018, +0.010] | yes |
| eval-set GRPO | SP_MOUSE | 41 | −0.022 [−0.041, −0.003] | yes |

Direction-matches (3/4 vs 2/4 SP classes for GRPO vs base) on a soft-token signal
that GRPO did not directly reward. See `figures/cai_yeast_minus_ecoli_forest.pdf`.

## Methodology notes

- **MFE per-seq** is computed with ViennaRNA `RNA.fold` under DNA Mathews 2004 parameters.
  Sequences ≤ 5 kb are folded full-length; longer sequences are summarised as the mean
  MFE density across **5 random 2 kb windows** with a per-sequence deterministic seed.
  Earlier `extra_metrics.json` uses the older 200-sample × 500 bp-window method;
  the per-seq parquet uses the newer thorough method, so summary numbers will not match
  exactly between the two.
- **Evo 2** uses `evo2_7b_base`. Sequences > 8 kb are scored on sliding windows with
  50 % overlap, perplexities averaged on the log scale. Embeddings (when emitted by the
  Evo 2 forward call) are mean-pooled across positions to a single (4096,) vector.
  The 2026-04-27 base run produced PPL only — embeddings were not returned by the
  current `evo2.Evo2.__call__` signature; the 2026-04-08 GRPO run did get them.

## Generation Method

Uses the model's optimized `generate_simple()` with KV cache. Prompts sampled from
training set (5,124 unique prompt templates). Each prompt is a sequence of special tokens
encoding plasmid properties (vector type, species, resistance markers, origins,
promoters, etc.) followed by `<SEQ>`.

## Tools

| Tool | Version | Purpose |
|------|---------|---------|
| pLannotate | 1.2.2 | Feature annotation (snapgene DB 2021-11) |
| Prodigal | 2.6.3 | ORF prediction |
| dustmasker (BLAST+) | 2.17.0 | Low-complexity masking |
| sourmash | 4.3.0 | Memorization check (k-mer Jaccard) |
| minimap2 | 2.30 | Follow-up alignment |
| LightGBM | 4.6.0 | Discriminator |
| ViennaRNA | 2.x | MFE density (DNA Mathews 2004 params) |
| Evo 2 | `evo2_7b_base` (StripedHyena) | Cross-model perplexity + embeddings |

## Phase 1: First-Light Sweep (2026-04-08)

Grid search over 12 sampling configs (3 temperatures × 2 top_p × 2 repetition_penalty),
50 sequences per cell = 600 total generations.

**Winner: temp=0.7, top_p=0.95, rep_pen=1.0**
- 70.8% median pLannotate coverage
- 22.1 mean features per sequence
- 7.6 kb mean length, 50.5% GC

## Reproducing the figures

The paper figure script lives at `paper/generate_figures.py` (in the
PlasmidLLM repo). Populate a local cache from this bucket via

    hf buckets sync hf://buckets/McClain/PlasmidLMEval /tmp/plasmidlm_eval_cache

then `uv run --script paper/generate_figures.py`.