Spaces:

TabArena
/

leaderboard

Running

App Files Files Community
leaderboard / website_texts.py
LennartPurucker
Claude Opus 5.5
Tour: button names show as pills, so the reader can match them to the screen
68988c6 1 day ago
Raw History Blame Contribute Delete
77.2 kB
from __future__ import annotations

# --- Page chrome: the top bar and the dialogs it opens ----------------------
# Built by views.make_page_chrome. The bar sits above the page tabs on every page, so its copy
# names the ecosystem; a benchmark's own links (paper, datasets, citation) belong to its page.
TOPBAR_LOGO_URL = "https://avatars.githubusercontent.com/u/210855230"
TOPBAR_BRAND = "TabArena"
TOPBAR_BRAND_SUFFIX = "Ecosystem"
TOPBAR_TAGLINE = "Living benchmarks for tabular ML"
# The brand's hover text, where there is room for the whole sentence.
TOPBAR_TAGLINE_LONG = "Living benchmarks and leaderboards for machine learning on tabular data"
TOPBAR_CODE_URL = "https://tabarena.ai/code"
# Where the benchmark's datasets are collected and curated.
TOPBAR_DATA_URL = "https://github.com/TabArena/data-foundry"
# Page slug -> (emoji, label, hover text). A page missing here shows its tab name.
TOPBAR_NAV = {
    "tabarena": ("🥇", "TabArena-v0.1", "The living benchmark for predictive machine learning on IID tabular data"),
    "beyondarena": (
        "🧭",
        "BeyondArena",
        "Beyond the IID assumption: IID, temporal and grouped splits across dataset sizes",
    ),
    "ramanbench": ("🔬", "RamanBench", "Machine learning on Raman spectra, built on TabArena"),
    "timeseries": ("⏳", "Time-series", "Where to find the time-series benchmarks"),
    "contribute": ("➕", "Your Benchmark?", "Bring your benchmark into the TabArena ecosystem"),
}
# The two groups of other benchmarks in the site's bar, as (label, hover text): those TabArena
# verified and approved, partly built on its infrastructure (RamanBench), and those outside it
# (the links below, and the Time-series signposts).
TOPBAR_GROUPS = {
    "built": ("Built on TabArena", "Verified and approved by TabArena, partly built on its infrastructure."),
    "outside": (
        "Outside TabArena",
        "Not held to the TabArena protocol and built outside it: academic benchmarks with their own standards.",
    ),
}
# Benchmarks outside TabArena that the bar links to directly, with no page here, as (logo, label, url,
# hover text); the logo is its file in assets/logos, with a `-dark` twin for the dark theme.
TOPBAR_OUTSIDE_LINKS = [
    (
        "strable.svg",
        "STRABLE",
        "https://huggingface.co/spaces/inria-soda/strable-leaderboard",
        "STRABLE: tables with strings. Opens its own leaderboard.",
    ),
    (
        "multabench.svg",
        "MulTaBench",
        "https://huggingface.co/spaces/multabench/leaderboard",
        "MulTaBench: tables with text and images. Opens its own leaderboard.",
    ),
]
# The theme switch's label, shown beside its icon in the opened rail.
TOPBAR_THEME_LABEL = "Theme"

# --- The tour (main.taStartTour) ---------------------------------------------
# One walk through the site, started from the rail, from the Learn more row or by a `?tour` link.
# It visits BeyondArena once, near the start, and runs the rest on TabArena; at the end the page
# is put back as it was. Each step is (key, title, text), in order; main.TA_TOUR_STEPS names what
# each key points at and does. A button's name in the text is written [[name]] and shows as a small
# pill, so the reader can match it to the screen.
TOUR_START = "Take the tour"
TOUR_START_HINT = "A short walk through the page: what each part shows and what you can click"
TOUR_LABELS = {
    "aria": "Guided tour",
    "count": "Tour {i}/{n}",
    "back": "Back",
    "next": "Next",
    "finish": "Finish",
    "play": "Play",
    "pause": "Pause",
    "exit": "Exit",
}
# The note a first-time visitor gets, once per browser (main.taTour): it offers the tour.
TOUR_NUDGE = {
    "head": "New here?",
    "text": "A one-minute tour shows what you can click.",
    "start": "Take the tour",
    "dismiss": "No thanks",
}
TOUR_STEPS = [
    (
        "rail",
        "Benchmarks",
        "This rail switches between benchmarks. [[TabArena-v0.1]] is the benchmark for IID tabular data and "
        "[[BeyondArena]] goes beyond it. Opened, it also lists RamanBench, built on TabArena, then benchmarks "
        "outside TabArena that are not held to its protocol (STRABLE, MulTaBench, time series), and how to "
        "add your own.",
    ),
    (
        "beyondarena",
        "BeyondArena",
        "BeyondArena benchmarks tabular data beyond the IID assumption: temporal and grouped splits across "
        "dataset sizes and feature types. It has the same views, so everything this tour shows works there "
        "too. [[Next]] comes back to TabArena.",
    ),
    (
        "hero",
        "TabArena-v0.1",
        "What the benchmark measures, in one sentence. [[Learn more]] holds the datasets, the models and "
        "systems, the metrics and the API, and [[Why this benchmark?]] opens its selling points.",
    ),
    (
        "tabs",
        "Views",
        "Five views of the same leaderboard. [[Summary]] is the overview and [[Leaderboard]] the full table; "
        "the others go deeper. Each view has its own link.",
    ),
    (
        "customize",
        "Customize",
        "Pick the datasets, who competes and what you care about. Every number on the page follows your choice.",
    ),
    (
        "shortcuts",
        "Shortcuts",
        "One-click starting points, such as [[Add systems]] or [[Regression only]]. A second click undoes one.",
    ),
    (
        "elo",
        "Elo",
        "How often a method beats the others, as a rating: higher is better, and 400 points is about a 91% "
        "win rate. The small numbers are its 95% interval, medals mark each column's top three, and a click "
        "on any header sorts by it.",
    ),
    (
        "breakdowns",
        "Costs and breakdowns",
        "The bars after Elo and Improvability are the median train and predict time per 1,000 rows. Fold out "
        "a group to see where a method is strong: [[By Task]], [[By Dataset Size]] or [[By Target]].",
    ),
    (
        "marks",
        "Marks",
        "After a name: ✔️ a verified implementation, $ a license that does not permit commercial use, and a "
        "red leakage warning where test data is proven to have leaked into a method's training. Hover any "
        "mark for details.",
    ),
    (
        "follow",
        "Follow a method",
        "Pick methods to mark them in every table and figure on the page, or click [[◎]] on a row.",
    ),
    (
        "datasets",
        "Per dataset",
        "The benchmark one dataset at a time: pick a method and see where it wins and where it loses.",
    ),
    (
        "time",
        "Quality vs time",
        "The Pareto fronts: the best quality for each time budget, for prediction, training or both. That is "
        "the tour; to put your own model on the board, see [[Benchmark your model]].",
    ),
]
RELEASE_UPDATED_LABEL = "Updated"
VERSION_HISTORY_TITLE = "Version history"
VERSION_HISTORY_HINT = "Version history: what changed in each release"
VERSION_HISTORY_SUB = "Every change to the leaderboard, newest first."

# The views of a benchmark page, as tabs at the top of its controls (views.view_bar), as (key, tab
# label). The first, the Summary, is the default, and `?view=<key>` opens another.
VIEW_TABS = [
    ("summary", "Summary"),
    ("leaderboard", "Leaderboard"),
    ("time", "Quality vs time"),
    ("h2h", "Head to head"),
    ("datasets", "Per dataset"),
]
# A tab's hover text: what its view shows.
VIEW_HINTS = {
    "summary": "Every method on every task and dataset subset at once, with its cost: who is strong "
    "everywhere and who shines on one subset. Per subset, with imputation, over all repeats.",
    "leaderboard": "The Elo ratings, then every number in the full table, on the datasets selected below.",
    "time": "The quality against training and prediction time, and what tuning buys.",
    "h2h": "The win rate of every pair of methods, on the datasets selected below.",
    "datasets": "Each method's rank and gap to the best on each dataset, as the chart and its ranking.",
}
# BeyondArena's hover texts where its views differ from TabArena's.
BEYOND_VIEW_HINTS = {
    "summary": "Best result per model and per model family on every BeyondArena subset: which methods "
    "hold up beyond the IID assumption.",
}

# --- The page heroes (views.make_tabarena_hero, views.make_beyond_hero) -------
# What the benchmark is, in one sentence (`**…**` is bold), heading the hero.
HERO_LEAD = (
    "**TabArena-v0.1** is a living benchmark for predictive machine learning on IID tabular data, built to "
    "measure the peak performance of model-specific pipelines."
)
# The paper and its citation share one split button, as (emoji, label) halves on every page.
HERO_PAPER = ("📄", "Paper")
HERO_CITE = ("📙", "Cite")
HERO_TEAM_LABEL = "Team"
# The link that ends the hero's sentence and opens the band of selling points under it.
HERO_WHY = "Why this benchmark?"
HERO_PAPER_URL = "https://tabarena.ai/paper-tabular-ml-iid-study"
# The call to submit, as (label, link): the highlighted pill that ends the Learn more row.
HERO_CTA = ("Benchmark your model", "https://tabarena.ai/code-examples")
# The Learn more pills' hover texts, as {topic: text}. The fields in braces are read from the data when
# the page is built (views.hero_facts).
HERO_HINTS = {
    "datasets": "{datasets} datasets, hand-picked from 1,053 real-world candidates for IID tabular data",
    "models": (
        "{entrants}, run on the same protocol and budget, each scored at its default, tuned, and as an "
        "ensemble of tuned configurations. {verified} of the {models} models are verified by their authors "
        "or the maintainers"
    ),
}
# The band HERO_WHY opens under the hero's sentence (views._page_hero): the selling points, as
# (value, label, line, hover text, target). The fields in braces are read from the data and the
# version history when the page is built (views.hero_facts). A click opens the target: a Learn more
# topic ("doc:<topic>"), the version history ("history") or a link.
HERO_STATS = [
    (
        "{datasets}",
        "curated datasets",
        "from 1,053 candidates",
        "Hand-picked from 1,053 real-world candidates for IID tabular data",
        "doc:datasets",
    ),
    (
        "{methods}",
        "{methods_label}",
        "each tuned to its peak",
        (
            "{entrants}, run on the same protocol and budget. Every model is scored at its default, "
            "tuned, and as an ensemble of tuned configurations"
        ),
        "doc:models",
    ),
    (
        "{releases}",
        "live updates",
        "{fresh}",
        "{releases} releases since {since}, as new models join. Last update: {updated}",
        "history",
    ),
    (
        "Open",
        "source & data",
        "verified implementations",
        (
            "The code, the datasets and every prediction are public, and {verified} of the {models} "
            "models are verified by their authors or the maintainers"
        ),
        TOPBAR_CODE_URL,
    ),
]
# The third selling point's line on TabArena: the newest entrant in the version history, or the date of the
# last update when no release names one.
HERO_FRESH_NEWEST = "newest: {newest}"
HERO_FRESH_UPDATED = "last update {updated}"

# The "Learn more" row: each topic opens one dialog, which switches between them.
LEARN_MORE_LABEL = "Learn more"

# The Team chip's dialog. The core maintainers as (name, GitHub login), in the paper's author order;
# BeyondArena has its own (BEYOND_TEAM).
TEAM_TITLE = "The TabArena team"
TEAM_SUB = "Maintained in the open by people from different, competing institutions."
TEAM = [
    ("Nick Erickson", "Innixma"),
    ("Lennart Purucker", "LennartPurucker"),
    ("Andrej Tschalzev", "atschalz"),
    ("David Holzmüller", "dholzmueller"),
]
TEAM_PRIMARY_CONTACT = "LennartPurucker"
TEAM_PRIMARY_LABEL = "Primary contact"
TEAM_CONTACT = """
- **Questions about the leaderboard?** Ask in the Space's [Community tab](https://huggingface.co/spaces/TabArena/leaderboard/discussions).
- **About the code, a model or a result?** Open an issue on [GitHub](https://tabarena.ai/code).
- **Anything else?** Email the team at **mail@tabarena.ai**, or reach out to our primary contact, **Lennart Purucker**, directly.
"""

# Conflict-of-interest statement, a dialog opened from the top bar's "Our COI" button (see
# views.make_page_chrome). Condenses our public position: it names the conflict plainly, then
# makes the case (open code, open evidence, competing institutions) that it does not affect the
# validity of the results.
COI_HTML = """
<div class="ta-dialog-widget">
  <input type="checkbox" id="coi-toggle" class="ta-dialog-toggle" hidden>
  <div class="ta-dialog-overlay">
    <div class="ta-dialog" role="dialog" aria-modal="true" aria-label="Conflict of interest statement">
      <button type="button" class="ta-dialog-x" title="Close" aria-label="Close" onclick="taCloseDialogs()">×</button>
      <div class="ta-dialog-body">
      <h2>Our Conflict of Interest</h2>
      <div class="coi-tldr"><b>TL;DR:</b> TabArena maintainers also develop models and are affiliated with
      commercial institutions. This creates a conflict of interest. We do not hide it, and we believe (and let
      anyone verify) that it does not affect the validity or scientific rigor of the results shown here.</div>
      <h3>Where the conflict lies</h3>
      <ul>
        <li><b>Maintainers vs. model developers.</b> Several of us develop tabular models that appear on this
        leaderboard. Building the benchmark and competing on it at the same time is a tension we can't fully
        remove (it is also documented in the <i>Competing Interests</i> section of our NeurIPS paper).</li>
        <li><b>Science vs. industry.</b> Some maintainers are employed by commercial institutions. We are also
        open-source researchers and PhD students maintaining TabArena in the open.</li>
      </ul>
      <p>We won't pretend that simply telling you we act without bias settles the question; it doesn't, and you
      are right to stay skeptical. And we'll say it outright: maintainers building a benchmark that features
      their own models is not, by itself, a prudent arrangement.</p>
      <h3>Why we believe the results still stand</h3>
      <p>Rather than ask you to trust us, we try to make trust unnecessary by letting anyone check everything:</p>
      <ul>
        <li><b>Everything is open source:</b> the benchmarking code, the leaderboard, the plots, the model
        implementations, and the models themselves.</li>
        <li><b>So is the evidence:</b> we publish the raw predictions, hardware details, and exact software
        environments. Anyone can reproduce our numbers, hunt for errors, submit their own model, and call us
        out.</li>
      </ul>
      <p>This puts the burden of proof on the other side: we hand any critic everything they would need to prove
      we did something wrong. Almost no one will check all of it, but the point is that anyone can.</p>
      <h3>Competing interests as checks and balances</h3>
      <p>TabArena is maintained, on purpose, by people from <i>different and competing</i> institutions, whose
      interests pull against each other. That way no single model or company sets the rules unchallenged, and we
      keep widening that mix as the benchmark grows.</p>
      <h3>What we are still working on</h3>
      <p>TabArena is young and still changing fast. The biggest thing we're missing is governance: we'd like
      proper open-source governance, and we have a TabArena organization and are heading that way, but right now
      we're too few people with too little time to set it up. We bring in more of the community with every
      release, and we'll keep adjusting.</p>
      <a class="coi-cta" href="https://tabarena.ai/code" target="_blank" rel="noopener noreferrer">💬 Concerns? Open a public issue for a public discussion ↗</a>
      </div>
    </div>
  </div>
</div>
"""

OVERVIEW_DATASETS = """
The leaderboard is built on a manually curated collection of **51 classification and regression datasets**
for independent and identically distributed (IID) tabular data, hand-picked from 1,053 real-world candidates.
They span the small-to-medium data regime and were chosen to reflect a wide range of real-world predictive
machine learning use cases. They are collected and curated in
[Data Foundry](https://github.com/TabArena/data-foundry).
**Subsets:** *Customize your leaderboard* narrows the benchmark to one task type, dataset size or class
balance, and its settings add the imputed models or switch to TabArena-Lite.
"""
OVERVIEW_MODELS = """
TabArena ranks two kinds of entrant, and the difference is the pipeline.
- 🤖 **Models** are single methods, and TabArena owns the pipeline around them. It tunes each one
  under a shared protocol on the same compute budget as every other model, and reports three
  regimes: **default**, **tuned**, and an **ensemble of tuned** configurations. Each sits in a
  tested, real-world pipeline built by the maintainers and, where possible, with the model's
  original authors.
- 📊 **Systems** own their pipeline end to end. A system decides what to fit, how much to tune it
  (if at all), and how to combine the result, all inside one budget; TabArena hands it the data
  and the constraints and nothing else. AutoGluon is the familiar example; TabFM+, agents and
  hosted APIs are others.
Both are measured the same way and held to the same budget, so the comparison is fair on final
performance. What differs is how much the setup is controlled. A model's score isolates one
method under a fixed pipeline; a system is free to step outside those constraints and do whatever
works, so its score reflects the whole package without saying which part earned it. That makes
systems the better guide to what you should run, and models the better basis for a research
claim. Use **Who's competing?** at the top of the page to choose which of them appear.
Systems carry tags for the things that might rule them out for you. **with LLMs** means an LLM
is somewhere in the loop, possibly as an agent, so the result depends on a model we cannot
inspect and that may already have seen the test data. 🔒 **closed-source API** means the system
runs on its provider's servers. We cannot see what it does with the data we send, test features
included, and it can change between runs; its train and predict times are not comparable with the
models'. Do not trust these results by default.
**Verified:** A ✔️ marks an implementation confirmed by the original authors or the TabArena
maintainers. Established, stable models (XGBoost, LightGBM, CatBoost, Random Forests, the
baselines) count as verified once the maintainers confirm the implementation. Treat unverified or
very recent entries with more caution. For a closed-source API system, ✔️ covers the client and
how TabArena calls the service; the service itself cannot be checked.
"""
OVERVIEW_METRICS = """
Each model is scored on every dataset with a **metric** suited to the task: **ROC AUC** for binary
classification, **log-loss** for multiclass classification, **RMSE** for regression. Those per-dataset
scores are then combined into a single leaderboard number by an **aggregation**. The leaderboards are ranked by the **Elo** aggregation, and we report several complementary
aggregations (Score, Improvability, Average & Harmonic Rank). Each is defined below, and the table shows the
same definition when you hover a column header.
**Imputation:** Toggle *Include imputed models* to add models that cannot run on every dataset, whether
because of the task type or the dataset size. Their missing results are filled in with a default
RandomForest's, which drags their score down. That is deliberate: a model that only covers part of the
benchmark should not rank as if it covered all of it.
**Repeats:** Toggle *TabArena-Lite* to score each experiment on a single repeat instead of several. It is
noisier than the full setting, but much cheaper to compute and usually close enough to be worth it.
"""
# --- The summary table, the Summary view (views.make_cross_subset_overview) ----------
# The search at the heading's right, which keeps the rows whose name holds its text.
OVERVIEW_FIND = "Find a method"
# The table's two metrics, as (headline column name, the name a breakdown's switch shows).
OVERVIEW_METRIC_NAMES = {"elo": ("Elo", "Elo"), "imp": ("Improvability (%)", "Improv.")}
# The table's rank column, and its folded breakdowns: the line under a stub's button, and the hover
# text of the button that folds an open breakdown again.
OVERVIEW_POS_HINT = "Rank in the column the table is sorted by"
OVERVIEW_STUB = "{n} subsets"
OVERVIEW_FOLD = "Fold these columns"

# --- The selector card (views.selector_card) ---------------------------------
SELECTOR_TITLE = "Customize"
SELECTOR_SUB = "Every number below is relative to who competes and which datasets count."
BEYOND_SELECTOR_SUB = "Every number below is relative to the datasets compared, so each subset is its own evaluation."
# The bar's shortcuts into the card (SELECTOR_TRY names the group for screen readers), as (label, hover
# text, row, value). A click picks that chip of the row and opens the card on it, a second click undoes
# it. In a row of toggles the value can be a tuple of chips turned on together; a switch names itself as
# the row and the state the shortcut puts it in as the value. Chips that cannot be picked yet are left out.
SELECTOR_TRY = "Make it yours"
SELECTOR_EXAMPLES = [
    (
        "Add systems",
        "Let every kind of system compete with the models, from AutoML frameworks such as AutoGluon to hosted APIs",
        "entrants",
        ("open", "llm", "api"),
    ),
    ("Regression only", "Rank on the regression datasets alone", "tasks", "regression"),
    ("Medium datasets", "Rank on the datasets with 10,001 to 100,000 training rows", "datasets", "medium"),
    ("Fastest predictions", "Lead with prediction speed rather than quality", "care", "infer"),
    ("Commercial licenses only", "Leave out the methods whose license forbids commercial use", "noncommercial", False),
    ("No leakage", "Leave out the methods with proven leakage of test data", "leakage", False),
]
BEYOND_SELECTOR_EXAMPLES = [
    ("Temporal splits", "Rank on the datasets split by time", "split", "temporal"),
    ("Text features", "Rank on the datasets with text features", "features", "text"),
    ("Large datasets", "Rank on the datasets with more than 100,000 rows", "size", "large"),
    ("Fastest predictions", "Lead with prediction speed rather than quality", "care", "infer"),
    ("Commercial licenses only", "Leave out the methods whose license forbids commercial use", "noncommercial", False),
    ("No leakage", "Leave out the methods with proven leakage of test data", "leakage", False),
]
# Row captions, and the captions of the chip groups within a row; the hover texts are in
# data_loading.AXIS_NOTES.
SELECTOR_CAPTIONS = {
    "entrants": "Who's competing",
    "tasks": "Task",
    "datasets": "Datasets",
    "tasks-classes": "by classes",
    "datasets-size": "by size",
    "datasets-target": "by target",
    "features-dim": "by dimensionality",
    "features-type": "by type",
    "care": "I care about",
    "metric": "Measured by",
    "settings": "Settings",
}
# The first chip of a row, which leaves that row unfiltered.
SELECTOR_ALL = "All"
SELECTOR_MODELS_ALWAYS = "Always on: TabArena is a model benchmark first."
# Ends the spec line of every BeyondArena leaderboard, since it holds for all of them.
BEYOND_SELECTOR_TAIL = ("core2k protocol",)
# What follows the number of datasets the selection covers: all of the page's, or part of them.
SELECTOR_TOTAL = ("datasets", "of {} datasets")
# A chip whose combination with the other rows has no dataset.
SELECTOR_NO_DATASETS = "No dataset matches this together with the other choices. Pick another chip in one of the rows."
# The spec line names the models-only pool this way.
SELECTOR_ALONE = "{} only"
# What each setting adds to the spec line when on and when off; an empty string adds nothing.
SELECTOR_SETTING_NOTES = {
    "imputed": ("with imputation", "without imputed models"),
    "lite": ("TabArena-Lite (one split)", ""),
    "noncommercial": ("", "commercial use only"),
    "leakage": ("", "without leakage warnings"),
}

# One warning per system category, shown under the selector while that category is competing, as
# (headline, details); the details open from the headline. They are not the same kind of caveat,
# and the styling says so: the open-source one is about how to *read* a result that is otherwise
# sound, so it gets the amber accent; the other two are about results that may not be sound at
# all, and keep the red.
WARNING_SYSTEMS = """
A system faces the same datasets, splits and metric as every model here, and it is held to the
same budget, so the comparison is fair on final performance: if a system scores higher, it really
did do better on this benchmark. What it gives up is the controlled setup. A model is one method
under a fixed pipeline, so its score isolates the method; a system is free to step outside those
constraints and do whatever works, so its score tells you what the whole package achieves without
telling you which part earned it.
"""

WARNING_WITH_LLM = """
These systems call an LLM, in some cases as an agent driving the whole pipeline. We cannot see
what that model was trained on, so we cannot rule out that it has already seen these datasets,
or read about them. A score that looks strong may be partly recall rather than prediction.
The model behind an API can also be replaced without notice, so the same system may not score
the same next month.
"""

WARNING_CLOSED_API = """
A closed-source API runs on its provider's servers. We send it the training data and the test
features, and record the predictions that come back. We cannot see what ran. A ✔️ on these rows
covers the implementation and how the benchmark uses the service; what runs behind the endpoint
cannot be verified.
- The provider receives the test features of public datasets whose labels are public too. A
  service could recognize a dataset and use labels it has already seen. Before a run, we probe
  the service for label lookup and for training on the test features. These probes catch the
  crude forms of both; passing them does not prove the service is clean.
- The version is whatever the provider reports, and the service can change between runs, so a
  result describes the service on the day we queried it.
- Train and predict times are not comparable with the models' times. A service may fit and
  predict in one call at prediction time. We then count that call as its train time too (marked
  †), since it holds the fit, and its predict time includes the fit. Both also carry network
  latency and data transfer, and run on hardware we do not control.
Read these rows as the provider's claim, checked as far as we can from the outside. For a
comparison you can rely on, use the models-only leaderboard.
"""
# --- Leakage warnings --------------------------------------------------------
# A method whose training is proven to have seen some of the benchmark's test data carries the
# warning: the artifacts' `Leakage` column (from `MethodMetadata.leakage_evidence` in the tabarena
# repo) links the evidence, and `LeakageDatasets` names the datasets it is proven on. Any dataset
# could be affected, so all its quality numbers may be too good. The Summary marks it
# (views.leakage_badge), the settings switch reads the leaderboard recomputed without it, and the
# API reports it. The badge's card says, with {proven} one of LEAKAGE_PROVEN:
LEAKAGE_METHOD_TEXT = "Leakage is proven on {proven}, and any other could be affected. All its results may be too good."
LEAKAGE_PROVEN = {0: "some test datasets", 1: "1 test dataset", 2: "{n} test datasets"}
# Above BeyondArena's overview figures while a flagged method is in them ({names}: who).
LEAKAGE_OVERVIEW_NOTE = (
    "⚠️ {names}: test data likely leaked into the training, so these results may be too good. Turn off "
    "*Include methods with a leakage warning* under *Customize* to see the overview computed without it."
)
# The badge after a flagged method's name, the mark before its quality numbers, and the key in the
# Method header with what its hover card says about the page as a whole ({n}: how many are shown).
LEAKAGE_BADGE = "Leakage"
LEAKAGE_KEY = "leakage warning"
LEAKAGE_NUMBER_HINT = "Leakage warning: this number may be too good"
LEAKAGE_PAGE_HEAD = {1: "1 method carries a leakage warning", 2: "{n} methods carry a leakage warning"}
LEAKAGE_PAGE_TEXT = (
    "Test data likely leaked into its training: proven on some datasets, possible on any. Its scores carry a "
    "red ! and win no medals."
)
LEAKAGE_PAGE_TEXT_MANY = (
    "Test data likely leaked into their training: proven on some datasets, possible on any. Their scores carry "
    "a red ! and win no medals."
)
LEAKAGE_METHOD_HEAD = "Leakage warning"
LEAKAGE_DATASETS = "Proven on: {}"
LEAKAGE_EVIDENCE = "Read the evidence ↗"
LEAKAGE_HIDE = "Hide them"
LEAKAGE_HIDE_ONE = "Hide it"

SYSTEM_WARNINGS = {
    "open": ("Systems and models are not the same kind of result.", WARNING_SYSTEMS),
    "llm": ("Results from LLM-based systems may be contaminated.", WARNING_WITH_LLM),
    "api": ("Do not trust closed-source API results by default.", WARNING_CLOSED_API),
}

# --- Section headings (views.section_head) ----------------------------------
# Ends the view bar's hover text after what the selection covers (views.lb_spec).
LB_HINT = "The leaderboard selected below: the datasets it covers, who competes and how it is computed."
BEYOND_LB_HINT = LB_HINT + " Every BeyondArena leaderboard is computed on the core2k protocol."
PER_DATASET_TITLE = "Dataset by dataset"
# The browser's header in the Datasets view. `{n}` is the number of datasets it covers.
PER_DATASET_CAPTION = "Where each method wins and loses, on each of the {n} datasets"
# What each figure panel shows, in its header beside the title.
FIGURE_CAPTIONS = {
    "overview": "Each method at its default, tuned, and tuned + ensembled settings",
    "infer": "Quality against prediction time",
    "total": "Quality against training plus prediction time",
    "train": "Quality against training time",
    "trajectories": "How quality grows as more configurations are tuned",
    "winrate": "How often each method beats each other one",
    "per_dataset": "Where each method wins and loses, one dataset at a time",
    "table": "Every method and variant on every metric: sort, filter, export",
}

# The full leaderboard table's card (views.make_leaderboard); main.taToggleTable flips the button.
TABLE_TITLE = "Full leaderboard table"
TABLE_SHOW = "Show the table ▾"
TABLE_HIDE = "Hide the table ▴"

TILE_EXPAND = "Expand the figure (Esc closes it)"

# "Follow a method": the picker in the leaderboard's heading and the ◎ on each overview row.
FOLLOW_LABEL = "◎ Follow a method"
FOLLOW_HINT = "Pick methods to mark in every figure and table of this leaderboard"
FOLLOW_FIND = "Find a method"
FOLLOW_ROW = "Follow {} in the figures below"

# The per-dataset browser. Shown while collapsed (the teaser), above it once opened (the
# intro), and instead of it when a leaderboard has no per-dataset artifact yet.
PER_DATASET_TEASER = (
    "Open this to browse the benchmark one dataset at a time: pick a method and see where it "
    "wins, where it loses, and how the whole field got there on each dataset."
)

SUBSET_MISSING = (
    "No results have been published for this combination yet. It is generated with the rest of the "
    "artifacts and appears here once the leaderboard has been regenerated."
)

BEYOND_CELL_EMPTY = (
    "No BeyondArena dataset matches every filter selected above. Set one of the rows back to its "
    "\"All\" chip to widen the selection."
)

PER_DATASET_MISSING = (
    "Per-dataset results have not been published for this leaderboard yet. They are generated "
    "with the rest of the artifacts and appear here once this combination has been rerun."
)

CITATION_BUTTON_LABEL = (
    "If you use TabArena or the leaderboard in your research please cite the following:"
)
CITATION_BUTTON_TEXT = r"""@inproceedings{erickson2025tabarena,
  title     = {TabArena: A Living Benchmark for Machine Learning on Tabular Data},
  author    = {Erickson, Nick and Purucker, Lennart and Tschalzev, Andrej and Holzm{\"u}ller, David and Desai, Prateek Mutalik and Salinas, David and Hutter, Frank},
  booktitle = {Proceedings of the 39th Conference on Neural Information Processing Systems (NeurIPS)},
  year      = {2025},
  url       = {https://arxiv.org/abs/2506.16791}
}
"""

# Single source of truth for the definitions of the table's columns: the header tooltips and the
# list in the Metrics topic of Learn more (views.metric_definitions_html).
METRICS_HEADING = "The leaderboard's columns"
METRICS = [
    {
        "name": "🏆 Elo (ranking aggregation)",
        "details": (
            "A pairwise comparison-based rating: each model's rating predicts its expected win "
            "probability against others, with a 400-point gap corresponding to a 10:1 (~91%) "
            "expected win rate. We calibrate 1000 Elo to a default Random Forest and bootstrap "
            "95% confidence intervals. Elo is computed from ROC AUC (binary), log-loss "
            "(multiclass), and RMSE (regression)."
        ),
        "why": (
            "Elo aggregates many head-to-head comparisons into a single interpretable scale that "
            "is robust to the choice of error normalization and to a few extreme datasets, which "
            "is why it is our primary ranking aggregation."
        ),
    },
    {
        "name": "📊 Score",
        "details": (
            "Following TabRepo, a normalized score: we linearly rescale the error so the best "
            "method scores 1 and the median method scores 0 (negative values clipped to 0), then "
            "average across datasets."
        ),
        "why": (
            "Gives an intuitive 0–1 sense of how close a model is to the best, complementing "
            "Elo's purely relative scale."
        ),
    },
    {
        "name": "📉 Improvability (%)",
        "details": (
            "How many percent lower the best method's error is than this method's, on a dataset, "
            "averaged over datasets: (err − best_err) / err × 100%. Always between 0% and 100%."
        ),
        "why": (
            "Directly answers 'how much could I still gain by switching to the best model here?' "
            "in the error-relative terms practitioners care about."
        ),
    },
    {
        "name": "🔢 Average Rank",
        "details": "Per-dataset rank of each method (lower is better), averaged across datasets.",
        "why": (
            "Simple and familiar, but sensitive to ties and treats every dataset equally "
            "regardless of how large the performance gap is."
        ),
    },
    {
        "name": "🎯 Harmonic Rank",
        "details": (
            "The harmonic mean of per-dataset ranks, 1 / ((1/N) · Σ 1/rankᵢ), which more strongly "
            "rewards being very good on some datasets."
        ),
        "why": (
            "Favors models that are sometimes excellent (and thus useful inside an ensemble or "
            "portfolio) over models that are uniformly mediocre."
        ),
    },
    {
        "name": "⏱️ Train / Predict Time (s/1K)",
        "details": (
            "Median training and prediction time in seconds per 1,000 samples, measured on the "
            "hardware shown in the Hardware column."
        ),
        "why": (
            "Peak accuracy is not free. These columns let you trade quality against training "
            "cost and inference latency for your own deployment."
        ),
    },
    {
        "name": "✔️ Verified",
        "details": (
            "✔️ marks models whose implementation was verified by the original authors or the "
            "TabArena maintainers; ➖ marks contributed-but-unverified implementations."
        ),
        "why": (
            "Recent or unverified results should be read with more caution; we surface this so "
            "rankings are not taken at face value."
        ),
    },
    {
        "name": "🧩 Imputed (%)",
        "details": (
            "Percentage of datasets where a model could not run (due to task or dataset-size "
            "constraints) and was imputed with a default Random Forest's performance."
        ),
        "why": (
            "Imputation makes benchmark coverage part of the score rather than hiding it: a model "
            "is penalized for not being able to run everywhere."
        ),
    },
]

# The MCP & API topic of the Learn more dialog: on TabArena how to choose a model automatically,
# then (on both pages) how to read the numbers from a script or an AI assistant.
AGENTS_TOPIC = ("agents", "MCP & API")
AGENTS_CHOOSING = """
#### Choosing a model automatically
TabArena is designed so that an automated system, an LLM agent included, can pick a tabular model from
evidence rather than guesswork. The leaderboards expose exactly the signals such a system needs:
accuracy (**Elo** / **Score**), robustness (**Average** & **Harmonic Rank**), and cost
(**Train / Predict Time**). A few heuristics for choosing automatically:
- **No tuning budget?** Compare the **(default)** variants, which show what each model achieves out-of-the-box,
  with no hyperparameter search.
- **Small datasets?** Check the **Small** subset to see which models lead in that regime.
- **Medium / larger datasets?** Check the **Medium** subset for the strongest models at that scale.
- **Task-specific?** Use the **Classification**, **Regression**, **Binary**, and **Multiclass** subsets to find the
  leaders for your task type.
- **Latency-bound deployment?** Sort by **Median Predict Time (s/1K)** to weigh accuracy against inference cost.
- **Want one robust pipeline instead of a single model?** Switch *Who's competing* to include systems and look at **AutoGluon**.
- **Production trust:** prefer **Verified ✔️** models, and treat unverified or very recent entries as provisional.
These are starting heuristics rather than guarantees, so always validate on your own data. These aggregations and their
trade-offs are motivated in our [paper](https://arxiv.org/abs/2506.16791).
"""
BEYOND_AGENTS_LEAD = """
Every BeyondArena leaderboard can be read by a script or an AI assistant as well as on this page: as a CSV
in the Space repo, or over MCP.
"""
AGENTS_API = """
#### Reading the numbers programmatically
Every leaderboard on this page is a CSV in the Space repo, so a script or agent can read the same values the
table shows without scraping it:
```bash
curl -sL https://huggingface.co/spaces/TabArena/leaderboard/resolve/main/data/entrants_models/imputation_yes/splits_all/tasks_all/datasets_all/website_leaderboard.csv
```
Swap the path segments for another subset (`entrants_{models,open,llm,api,open_llm,open_api,llm_api,open_llm_api}` /
`imputation_{yes,no}` / `splits_{all,lite}` / `tasks_{all,classification,regression,binary,multiclass}` /
`datasets_{all,small,medium,balanced,imbalanced,extreme}`); BeyondArena lives
under `data_beyondarena/subsets/split_{all,random,temporal,grouped}/size_{all,tiny,small,medium,large}/`
`features_{all,low-dim,high-dim,text,high-cardinality}/tasks_{...}/`, with no folder for a combination that
no dataset falls into.
The per-dataset, per-split numbers behind these tables (one row per dataset, split and method) are not
hosted. Generate them with
[`examples/plots/run_export_results_per_split.py`](https://github.com/autogluon/tabarena/blob/main/examples/plots/run_export_results_per_split.py)
from the TabArena repository, which writes them as `results_per_split.csv`; swap in `BeyondArenaContext` for
BeyondArena.
#### Connecting an AI assistant (MCP)
This Space is also an **MCP server**, so an assistant can query the leaderboard as a tool instead of you
pasting numbers into a chat. Four tools are exposed:
- `list_leaderboards`: the available benchmarks, their valid subset values, and the keys every record carries.
  Worth calling first.
- `get_tabarena_leaderboard`: ranked results for one IID subset (tasks, datasets, imputation, splits).
- `get_beyondarena_leaderboard`: the same for BeyondArena, filtered by split, size, features and task type
  in any combination.
- `get_pareto_frontier`: the accuracy-versus-time trade-off. Ask this rather than reading the top row: the
  highest-Elo model is often far slower than one just behind it, and this returns the outright best, the
  models nothing beats on both axes at once, and the best model that fits a train- or predict-time budget.
All four take a `kind` of `models` (the default), `systems`, or `all`: individual models like TabPFN, whole
AutoML systems like AutoGluon, or both.
**Claude Code**: add the server once, then use it in any session:
```bash
claude mcp add --transport http tabarena https://tabarena-leaderboard.hf.space/gradio_api/mcp/
```
Add `--scope user` to make it available across all your projects. `claude mcp list` shows `✔ Connected`
when it worked, and `/mcp` inside a session lists the tools. The trailing slash on the URL matters.
**Claude Desktop and claude.ai**: add the same URL as a custom connector in settings.
Any other MCP client works too; the endpoint speaks streamable HTTP. Then just ask, for example:
*"Using TabArena, what's the best tabular model for a 5k-row regression dataset if inference has to stay
under 0.5 s per 1000 rows?"*
"""

# --- BeyondArena tab -------------------------------------------------------
# BeyondArena is the second internal leaderboard: the first unified benchmark for
# tabular data that goes beyond the IID assumption. Copy mirrors the TabArena
# HERO_* / OVERVIEW_* / CITATION_* structure, plus a Subsets topic, with BeyondArena specifics.
BEYOND_HERO_LEAD = (
    "**BeyondArena** is the first unified benchmark for tabular data beyond the IID assumption, with "
    "IID, temporal and grouped splits across dataset sizes and feature types."
)
BEYOND_HERO_PAPER_URL = "https://arxiv.org/abs/2606.30410"
# As TEAM, in the BeyondArena paper's author order.
BEYOND_TEAM_TITLE = "The BeyondArena team"
BEYOND_TEAM = [
    ("Lennart Purucker", "LennartPurucker"),
    ("Andrej Tschalzev", "atschalz"),
    ("Nick Erickson", "Innixma"),
    ("Gioia Blayer", "gblayer"),
    ("David Holzmüller", "dholzmueller"),
    ("Alan Arazi", "alanarazi7"),
]
BEYOND_HERO_CTA = ("Benchmark your model", "https://github.com/autogluon/tabarena/tree/main/examples/beyondarena")
# As HERO_HINTS; the fields in braces come from views.beyond_hero_facts.
BEYOND_HERO_HINTS = {
    "datasets": (
        "{datasets} datasets, split at random, by time or by group, from tiny to large and with text or "
        "high-cardinality columns. Every leaderboard uses the core2k protocol: 2,000 splits over all of them"
    ),
    "subsets": "{subsets} subsets: pick one chip per row in the selector below, and each combination is its own leaderboard",
    "models": (
        "{entrants}, each scored at its default, tuned, and as an ensemble of tuned configurations. "
        "{verified} of the {models} models are verified by their authors or the maintainers"
    ),
}
# As HERO_STATS; the fields in braces come from views.beyond_hero_facts.
BEYOND_HERO_STATS = [
    (
        "{datasets}",
        "curated datasets",
        "IID, temporal & grouped splits",
        (
            "Split at random, by time or by group, from tiny to large and with text or high-cardinality "
            "columns. Every leaderboard uses the core2k protocol: 2,000 splits over all {datasets} datasets"
        ),
        "doc:datasets",
    ),
    (
        "{subsets}",
        "subsets to combine",
        "by split, size, features & task",
        "Pick one chip per row in the selector below, and each combination is its own leaderboard",
        "doc:subsets",
    ),
    (
        "{methods}",
        "{methods_label}",
        "each tuned to its peak",
        "{entrants}, each scored at its default, tuned, and as an ensemble of tuned configurations",
        "doc:models",
    ),
    (
        "Open",
        "source & data",
        "built on TabArena's code",
        (
            "The code, the datasets and every prediction are public, BeyondArena runs on TabArena's "
            "experiment and evaluation code, and {verified} of the {models} models are verified by their "
            "authors or the maintainers"
        ),
        TOPBAR_CODE_URL,
    ),
]

BEYOND_OVERVIEW_DATASETS = """
BeyondArena is built on a curated collection of **142 datasets** that deliberately go **beyond the IID
assumption**. Datasets span three **split regimes**: **IID / random**, **temporal** (train on the
past, test on the future), and **grouped** (disjoint groups between train and test): across a wide
range of **sizes** (tiny → large) and **feature dimensionalities** (incl. text and high-cardinality
categorical columns).
**Subsets:** Use the rows of *Customize your leaderboard* to focus on a split regime, size bucket,
feature subset or task type, and combine them freely. Every leaderboard is always computed on the
**core2k** protocol (2,000 splits over the 142 datasets, enough for stable rankings).
**Data Foundry:** The [datasets](https://huggingface.co/datasets/TabArena/BeyondArena) are curated and
distributed through [Data Foundry](https://github.com/TabArena/data-foundry), a framework for curating
tabular datasets introduced alongside the benchmark. They are downloaded and converted on demand.
"""
BEYOND_OVERVIEW_MODELS = """
Like TabArena, BeyondArena focuses on **model-specific pipelines**: each model is wrapped in a tested,
tuned real-world pipeline, evaluated with its **default** configuration, a **tuned** configuration,
and as an **ensemble of tuned** configurations. The benchmark spans tree-based models, deep-learning
models, and tabular **foundation models**.
**Verified models:** A ✔️ in the *Verified* column marks models whose implementation was confirmed by
the original authors or the maintainers. Treat unverified or very recent models with more caution.
**Benchmark your own model:** BeyondArena runs on the same experiment and evaluation code as TabArena;
the only swap is the context (`BeyondArenaContext`). The
[BeyondArena examples](https://github.com/autogluon/tabarena/tree/main/examples/beyondarena) compare
your model against the cached baselines.
"""
BEYOND_OVERVIEW_METRICS = """
**Metrics vs. aggregations.** Each model is scored on every dataset with a task-appropriate **metric**
(**ROC AUC** for binary classification, **log-loss** for multiclass, **RMSE** for regression). These
per-dataset scores are combined into a single leaderboard number by an **aggregation**. The
leaderboards are ranked by the **Elo** aggregation, alongside complementary aggregations (Score,
Improvability, Average & Harmonic Rank). Each is defined below, and the table shows the same
definition when you hover a column header.
**Protocol (core2k):** All results use BeyondArena's **core2k** subset: 2,000 splits over the 142
datasets, chosen by a cost-weighted allocation that starts from the first split of every dataset and adds
splits up to 2,000, so more splits go to the datasets where they are cheap and informative. Until
v0.1.9.5 the leaderboard used the 507-split **core** subset (see the appendix of the
[paper](https://arxiv.org/abs/2606.30410) for how it was computed).
**Imputation:** Models that cannot run on all datasets (due to task or dataset-size constraints) have
their missing results imputed with a default RandomForest, which penalizes them for not running
everywhere. A `[X% IMPUTED]` tag marks affected models.
"""

# The Subsets topic of the Learn more dialog, also opened from the overview figures (whose axes name
# the subsets) by this link.
BEYOND_SUBSETS_LINK = "What do the subsets mean?"
BEYOND_SUBSETS_EXPLAINER = """
BeyondArena is sliced into **curated subsets** so you can see how methods hold up *beyond* the IID
assumption, and specifically on the kinds of data where an average hides big differences.
The overview figure shows each subset on its own. In *Customize your leaderboard*, pick one chip per row
and the rows combine: **IID** with **Tiny** ranks the methods on the tiny IID datasets only, and a row's
**All** chip leaves it unfiltered. Each combination is its own evaluation on the **core2k**
protocol.
**🔀 Split regime, the beyond-IID axis** (how the train/test splits are drawn)
- **IID**: random splits, the classic i.i.d. assumption (train and test come from the same distribution).
- **Temporal**: time-based splits: train on the past, test on the future (distribution shift over time).
- **Grouped**: disjoint groups between train and test (e.g. different users/sites), so no group leaks across the split.
**📏 Dataset size** (by the number of training rows in a dataset's largest training split)
- **Tiny**: ≤ 1,000 rows &nbsp;·&nbsp; **Small**: 1,001–10,000 &nbsp;·&nbsp; **Medium**: 10,001–100,000 &nbsp;·&nbsp; **Large**: 100,001–1,000,000
**🧬 Features**
- **Low-dim**: ≤ 100 columns after preprocessing &nbsp;·&nbsp; **High-dim**: more than 100 columns.
- **Text**: datasets containing one or more text columns.
- **High-cardinality**: datasets containing one or more high-cardinality categorical columns.
**🎯 Task** (as on TabArena, with the same metrics)
- **Classification**: binary and multiclass together &nbsp;·&nbsp; **Binary**: ROC AUC &nbsp;·&nbsp; **Multiclass**: log-loss &nbsp;·&nbsp; **Regression**: RMSE
"""

BEYOND_CITATION_BUTTON_LABEL = "If you use BeyondArena in your research please cite the following:"
BEYOND_CITATION_BUTTON_TEXT = r"""@misc{purucker2026beyondiid,
  title         = {Beyond IID: How General Are Tabular Foundation Models, Really?},
  author        = {Purucker, Lennart and Tschalzev, Andrej and Erickson, Nick and Blayer, Gioia and Holzm{\"u}ller, David and Arazi, Alan and Pfefferle, Alexander and Tajjar, Mustafa and Varoquaux, Ga{\"e}l and Hutter, Frank},
  year          = {2026},
  eprint        = {2606.30410},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2606.30410}
}
"""

RAMANBENCH_TAGLINE = "Benchmarking machine learning on Raman spectra."
RAMANBENCH_BLURB = """
Raman spectra are a **special kind of tabular data**: each spectrum is a fixed-length vector of intensity
measurements across wavenumbers, which makes them a natural fit for the tabular ML methods benchmarked across
the TabArena Ecosystem.
> ℹ️ **Please note:** RamanBench is developed and maintained independently, not by the TabArena team. We have
> reviewed it, endorse its methodology, and are glad to feature it as a trusted part of the TabArena Ecosystem.
**RamanBench** is a domain-specific benchmark for machine learning on Raman spectroscopy data. Raman spectroscopy
is a well-established, non-invasive technique for inferring the composition and molecular properties of materials:
a sample is excited with a monochromatic laser beam, and the small fraction of light that is inelastically
scattered by the vibrations of its molecular bonds shifts in energy, encoding information about the molecular
structure. These spectra are used across material identification, bioprocess monitoring, medical diagnostics,
pharmaceutical quality control, and chemical process analysis, and machine learning has become central to
automating their analysis, across material classification, disease detection, and the quantitative prediction of
chemical concentrations. RamanBench brings a dedicated, reproducible leaderboard to this domain; explore the full
benchmark on Hugging Face, or read the paper for the methodology and results.
"""

# --- "Time-series?" signpost page ------------------------------------------
# TabArena is tabular ML; time series is a neighbouring but distinct field with
# its own dedicated benchmarks. This page introduces the domain, explains how it
# differs from IID tabular ML, highlights the TimeCopilot "Impermanent" living
# leaderboard, and points to the main forecasting and classification/regression
# benchmarks (the time-series related work referenced from BeyondArena).
TIMESERIES_TAGLINE = "Tabular ML lives here; time series can be found elsewhere."

TIMESERIES_INTRO = """
**TabArena** and **BeyondArena** benchmark machine learning on **tabular data**: including tabular data with
*temporal* relationships (BeyondArena's **`temporal`** subset). Dedicated **time-series** modelling is the
neighbouring world, split into two families:
- **Forecasting**: extrapolate a series *forward* in time (rolling-window evaluation, forecast horizons,
  careful leakage control).
- **Classification & regression**: map a *whole* sequence to a label or a continuous target.
> ℹ️ **Note:** these benchmarks are maintained independently, not by the TabArena team.
"""

TIMESERIES_FORECASTING = """
#### 📈 Time-series forecasting benchmarks & leaderboards
- **[Impermanent](https://impermanent.timecopilot.dev/)** (TimeCopilot): the live, leakage-free weekly
  leaderboard for *temporal generalization*. [Paper](https://arxiv.org/abs/2603.08707).
- **[GIFT-Eval](https://huggingface.co/spaces/Salesforce/GIFT-Eval)** (Salesforce): a broad general-purpose
  benchmark: 23 datasets, ~144k series and 177M points across 7 domains and 10 frequencies.
  [Paper](https://arxiv.org/abs/2410.10393).
- **[fev-bench](https://huggingface.co/spaces/autogluon/fev-bench)** (AutoGluon / Amazon): 100 realistic
  forecasting tasks over 7 domains, 46 of them with covariates.
  [Paper](https://arxiv.org/abs/2509.26468).
- **[TIME](https://huggingface.co/spaces/Real-TSF/TIME-leaderboard)** (ICML 2026): a next-generation,
  leakage-controlled benchmark for zero-shot foundation models: 50 freshly collected, human-vetted datasets and
  98 forecasting tasks. [Paper](https://arxiv.org/abs/2602.12147).
"""

TIMESERIES_CLASSREG = """
#### 🏷️ Time-series classification & regression
Here the task is to label or score an *entire* sequence rather than extend it:
- **[UCR Time Series Archive](https://arxiv.org/abs/1810.07758)**: the standard *univariate* classification
  archive.
- **[UEA multivariate archive](https://arxiv.org/abs/1811.00075)**: its *multivariate* counterpart, recently
  extended by **["The Multiverse of Time Series ML"](https://arxiv.org/abs/2603.20352)** (2026).
- **[Time Series Extrinsic Regression (TSER)](http://tseregression.org/)**: the Monash/UEA/UCR archive for
  predicting a continuous target from a whole series. [Paper](https://arxiv.org/abs/2006.10996).
"""

TIMESERIES_CLOSING = """
Building a time-series benchmark you would like featured here, or curious whether tabular foundation models
transfer to your temporal data? We would love to hear about it: see the **➕ Your Benchmark?** tab.
"""

# --- "Your Benchmark?" invite page -----------------------------------------
# A welcoming call-to-action inviting the community to bring their own benchmark
# into the TabArena Ecosystem — either an existing one we endorse & double-check
# (like RamanBench) or a brand-new one we help shape.
YOUR_BENCHMARK_TAGLINE = (
    "Your benchmark could live right here. Let's build this out together."
)

YOUR_BENCHMARK_INTRO = """
The **TabArena Ecosystem** is more than a single leaderboard. It is growing into a home for *living,
trustworthy benchmarks* for machine learning on tabular and tabular-like data, each one reproducible
and clearly documented.
Maybe you already maintain a benchmark and want it to reach more people. Maybe you are just starting to
think about building one and could use a hand. Either way there is a place for it here, and we are happy
to help you get there. 🤝
"""

# Two side-by-side "paths" describing how a benchmark can join the ecosystem.
# (icon, heading, body) — rendered as cards in render_invite_page().
YOUR_BENCHMARK_PATHS = [
    (
        "🔬",
        "You already have a benchmark",
        "Wonderful! If you maintain a reproducible benchmark for a tabular (or tabular-like) ML problem, "
        "we would love to feature it. We will go through the methodology with you, double-check the "
        "results, and once we are confident in it, endorse it and add it to the ecosystem as a "
        "trusted, clearly attributed benchmark, exactly as we did with RamanBench. You stay the owner "
        "and maintainer; we help with quality, visibility, and a shared home.",
    ),
    (
        "🌱",
        "You're building a new one",
        "Even better to talk early. Designing a fair, reproducible benchmark is genuinely hard, and we "
        "have learned a lot building TabArena. Reach out and we can help with dataset curation, "
        "evaluation protocols, metrics, leaderboard tooling, and fitting cleanly into the wider "
        "ecosystem, so your benchmark stands on solid ground from day one.",
    ),
]

YOUR_BENCHMARK_FIT = """
#### ✨ What makes a good fit
- **Tabular at heart**: classic tabular data, or data that is naturally represented as fixed-length
  feature vectors (like Raman spectra).
- **Reproducible**: clear datasets, splits, metrics, and an evaluation protocol that others can re-run.
- **Open & well-documented**: ideally backed by a paper or write-up and public code and data.
Not sure whether your idea fits? **Reach out anyway**: we are glad to think it through with you, and we
genuinely enjoy meeting people building benchmarks for the community.
"""

YOUR_BENCHMARK_CONTACT = """
#### 📬 Get in touch
Drop us a line and tell us about your benchmark: what it covers, where the data comes from, and what you
are hoping to achieve. No formal proposal needed to start a conversation; an email is plenty. You can email
the TabArena team, or reach out to our primary contact, **Lennart Purucker**, directly.
"""

# (label, href, variant) — contact buttons rendered as new-tab anchors.
YOUR_BENCHMARK_LINKS = [
    (
        "✉️ Email the TabArena team ↗",
        "mailto:mail@tabarena.ai?subject=New%20benchmark%20for%20the%20TabArena%20Ecosystem",
        "primary",
    ),
    (
        "Primary contact: Lennart Purucker ↗",
        "https://github.com/LennartPurucker",
        "secondary",
    ),
]

VERSION_HISTORY_BUTTON_TEXT = """
**Current Version: TabArena-v0.1.9.12**
The following details updates to the leaderboard (date format is YYYY/MM/DD):
* 2026/10/27-v0.1.9.12:
    * Leakage warnings. LimiX-2 is flagged: test data is proven to have leaked into its training on two
      datasets ([#648](https://github.com/autogluon/tabarena/issues/648)), and any other could be affected.
      The *Summary* marks its quality numbers with a red *!* and gives it no medals, and every chart
      marks its name. *Include methods with a leakage warning* under *Customize* (on by default)
      recomputes the whole leaderboard without it when turned off, as the systems toggles do.
    * A guided tour of the page: *Take the tour* in the rail or under *Learn more*.
    * The rail lists the other benchmarks when it opens: *Built on TabArena* (RamanBench) and *Outside
      TabArena*, with links to STRABLE and MulTaBench next to the time-series pointers.
    * AutoGluon shows its own logo.
    * The charts can rename methods, save a view, and take their own title and axis limits; the
      tuning trajectories offer every time axis, and *Per dataset* shows each dataset's size and winner.
* 2026/10/26-v0.1.9.11:
    * Each benchmark page shows one view at a time, picked from tabs: *Summary*, *Leaderboard*,
      *Quality vs time*, *Head to head* and *Per dataset*. Each view has its own link (`?view=…`).
    * *Summary* lists every method with its Elo, improvability and train and predict times as bars
      in the cells, and folds the results by task, dataset size and target into columns that
      open with a click.
    * *Customize*, one row above the views, picks the leaderboard, with shortcuts such as *Add
      systems* or *Regression only*.
    * The introduction is shorter, the benchmark links move to a rail on the left that opens on
      hover (on wide screens), and the floating bar at the bottom of the page is gone.
* 2026/10/24-v0.1.9.10:
    * New leaderboard design. A top bar links every TabArena benchmark, and one card picks the
      leaderboard, with the number of datasets on every choice; everything below follows it.
    * *All results at a glance* lists every method on every subset, with organization logos,
      sortable columns and family filters. *Dataset by dataset*, right under it, browses the
      benchmark one dataset at a time, with the list beside the chosen dataset.
    * *Your leaderboard* shows its figures as one grid, the figure you care about first. Follow a
      method, or click it in any chart, to mark it in every figure; hovering a method in one chart
      marks it in the others. The full table opens compact, with a toolbar and a columns menu.
    * The page is lighter: the interactive figures load as they come into view.
    * The last version before the redesign (v0.1.9.9) is archived, see the end of this list.
* 2026/10/21-v0.1.9.9:
    * Add new verified model: FTFM, a factorized tabular foundation model, submitted as its default
      configuration. It is classification-only, so it is imputed on the regression tasks.
* 2026/10/19-v0.1.9.8:
    * Add new verified model: TabSTAR-v1.1, the second generation of TabSTAR, submitted as its
      default configuration. It replaces TabSTAR (v1), which is removed from the leaderboard.
* 2026/10/13-v0.1.9.7:
    * Add new verified system: Xiaomi-TabLDM+, the enhanced inference path of Xiaomi-TabLDM. It
      fits its candidate ensemble weights on a hold-out split it carves from the training data,
      which the model protocol does not allow, so it competes as a system next to the
      Xiaomi-TabLDM model entry.
* 2026/10/11-v0.1.9.6:
    * Add new verified models: PrismBoost and LightPFN (classification only; its regression
      results are imputed).
    * Add the first closed-source API system: Chakra-Tab by YHat Labs, verified, in its
      medium (3-fold bagging) and full (8-fold bagging) presets. It competes when
      *🔒 Closed-source API systems* is selected under *Who's competing?*. The API fits and
      predicts in one call, so we count that call as its train time too (marked † in the
      tables), and its predict time includes the fit.
    * The warning shown with closed-source API systems and the Systems panel now say why their
      results should not be trusted by default, and that their train and predict times are not
      comparable with the models' (one call at prediction time, network latency, hardware we do
      not control). For these systems, ✔️ covers the client and how TabArena calls the
      service, not the service itself.
    * Systems are colored by category: open-source systems keep the System orange, systems with
      LLMs take a darker orange and closed-source API systems a rust shade, in every figure,
      table and legend.
    * A theme switch in the top-right corner picks Default, Light or Dark.
* 2026/10/06-v0.1.9.5:
    * Add new verified models: Kumo-Tabular, Kumo-Tabular-Medium and Kumo-Tabular-Small, on
      TabArena and BeyondArena.
    * BeyondArena moves from the 507-split *core* protocol to *core2k*: 2,000 splits over the 142
      datasets from a cost-weighted allocation, so more splits go to the datasets where they are
      cheap and informative. Every BeyondArena leaderboard, figure and filter combination is
      computed on core2k, so the ratings of all methods shift with it.
* 2026/10/02-v0.1.9.4:
    * BeyondArena: the split, size and feature selectors combine instead of resetting each
      other, and a new task selector (classification, regression, binary, multiclass) joins
      them. Each of the 354 combinations that contains at least one dataset is its own
      leaderboard.
    * The per-dataset browser understands the Classification task filter (binary and
      multiclass together); BeyondArena's opens on it now, TabArena's after its next refresh.
* 2026/09/28-v0.1.9.3:
    * Updated verified model: Linear, re-run with L1 regularization applied to classification.
      AutoGluon's linear model had fit the L1 configurations with L2, so the tuned and ensembled
      Linear rows gain Elo while the default configuration is unchanged.
    * BeyondArena: add nine foundation models (TabPFN-3.5, TabPFN-3.5-Fast, Causilo, LimiX-2,
      TabFM, EXAONE-Tabular, RealTabPFN-2.5, TabDPT-1.3, TabSwift) and the Linear re-run.
    * BeyondArena now uses the TabArena layout below its two overview figures: a selector for
      what you care about and for the split, size and feature subsets, then the interactive
      leaderboard overview, Pareto front, tuning trajectories, win-rate matrix, full table and
      per-dataset results. The per-dataset browser can filter by split type (IID, temporal,
      grouped). The overview figures label each family with its best model and give every model
      its own color.
    * Every page has its own link: add `?tab=beyondarena` (or `tabarena`, `ramanbench`,
      `timeseries`, `contribute`) to the address.
* 2026/09/22-v0.1.9.2:
    * Add new verified model: TabDPT-1.3 (TabDPT v1.3, which replaces TabDPT-Turbo as the
      installable TabDPT version; the TabDPT-Turbo entry stays).
* 2026/09/18-v0.1.9.1:
    * CatBoost is back on its previous run (July 2026). The re-run of v0.1.9 reproduced its Elo
      but measured fit times about 1.5 times higher, on different machines; it stays out until
      the CPU timing setup (which cores and how many) is settled and tested.
    * Per-dataset results: the tuning trajectories of the Linear model and KNN are shown again.
      They had been dropped along with the aggregate figure's baseline filter, so clicking them
      in the ranking did nothing even on datasets where they are competitive. A method without
      a published trajectory now says so in its tooltip.
    * A metric picked inside a plot explorer's edit view stays picked when switching back to
      the paper view; the page's own metric selector no longer overrides it.
* 2026/09/17-v0.1.9:
    * Updated verified models: Causilo, Mitra-v2, Xiaomi-TabLDM, TabFM, TabICLv2, EXAONE-Tabular,
      TabPFN-3 and Nori-30M, re-run under the new timing pipeline. Train and predict times no
      longer include the one-off warm-up (library imports, CUDA context, checkpoint loading) that
      the old measurement charged to the first fit, so the median train times of the foundation
      models drop two to four times; the Elo ratings are unchanged within their confidence
      intervals. Updated unverified model: TabSwift, from the same re-run.
    * Add new verified models: TabPFN-3.5 and TabPFN-3.5-Fast. Their fit times were re-measured
      without tabpfn's preprocessing worker pool, whose start-up had added a fixed 15 to 20 s to
      every fit of the first run; predictions and Elo are unchanged.
    * Updated verified model: CatBoost, re-run under the new timing pipeline with its default and 200
      random configurations on every split.
    * New *balanced*, *imbalanced* and *extreme* dataset subsets. Classification datasets are
      split on the class ratio of the largest to the smallest class (imbalanced from 10:1,
      extreme from 50:1), regression datasets on the skewness of the target.
    * Every method now declares its license and whether it permits commercial use. The new
      *Include non-commercial methods* toggle hides the research-only methods, and the table
      marks them.
    * The per-dataset results show each method's spread over the splits and which methods tie
      with the best; the explorers leave out the methods the page excludes.
* 2026/09/10-v0.1.8.3:
    * BeyondArena: the *tiny*, *small*, *medium* and *large* size subsets now bucket each
      dataset by its largest training split, as the paper does. The subsets were bucketed by
      the mean training split, which put two temporal datasets one bucket too low
      (*garments_worker_productivity* in tiny instead of small, *sf_permit_time* in medium
      instead of large). The size subsets now hold 52 / 38 / 30 / 22 datasets; the other
      BeyondArena subsets are unchanged.
* 2026/08/17-v0.1.8.2:
    * The *Performance across leaderboards* table now has a row per model variant, so default,
      tuned and tuned + ensembled each get their own line and the cost of tuning is on the page.
      Its new *One per model* toggle collapses each method to its best variant, as the win-rate
      matrix's button of the same name does.
    * Fixed the *Fit (s/1K)* and *Infer (s/1K)* columns of that table, which reported the cheapest
      variant of each model rather than the variant on the row. A row labelled
      *(tuned + ensembled)* showed its default's fit time, roughly 200x too low, because tuning
      fits 200 configurations where the default fits one.
* 2026/08/10-v0.1.8.1:
    * Updated verified model: ChimeraBoost, now at version 0.30.0.
* 2026/08/06-v0.1.8:
    * New *Per-dataset results* section: the benchmark one dataset at a time. Pick a contender
      and each row reports its rank and its gap to the best method on that dataset, with a ★
      marking where it landed among the whole field. Sort by any column, search or filter the
      list, then select a dataset for its own tuning trajectories and its full ranking; ← and
      → step through the list.
    * The full leaderboard table and the per-dataset browser are collapsed by default, so the
      page opens on the figures; the contents row above them opens either one. The two
      reference panels about the site itself now sit under an *Appendix* heading.
    * Systems are now their own class of entrant rather than a "reference pipeline" alongside
      the models. A system is a whole pipeline that picks, tunes and ensembles models inside
      its own budget: AutoGluon, TabFM+, an agent, a hosted API.
    * New *Who's competing?* control at the top of the page. Every leaderboard number is
      relative to the field (Elo is a pairwise rating, Improvability is the gap to the best
      entrant), so each setting is a separate published evaluation rather than a row filter.
      Models-only is the default, and it is now genuinely models-only: previously AutoGluon
      sat in the Elo pool while the figures hid it.
    * New *I care about* control, which leads with the figure that answers your question:
      best quality, fastest to train, or fastest to predict.
    * Systems carry tags for what you may need to rule out: 🤖 with LLMs and 🔒 closed-source
      API. See the new "Systems" info panel.
    * Add new verified systems: AutoGluon 1.6 (extreme, 4h) and AutoGluon 1.6
      (noncommercial, 4h). The noncommercial preset is extreme plus TabPFN-3, whose license
      does not permit commercial use.
    * Add new verified system: TabFM+. It ran before the July rerun that stopped reloading
      models from disk around inference, so its inference time reads high next to the models
      it sits beside on the Pareto front.
    * The win-rate matrix carries every tuning variant again, so you can compare a tuned
      model against another model's default. It still opens on one row per model.
    * The jump menu, the conflict-of-interest badge and the selection summary stay on screen
      on huggingface.co, and the conflict-of-interest popup opens where you are reading
      instead of sending you back to the top of the page.
    * An AI assistant connected to this Space now gets tool descriptions that say what
      TabArena measures and which questions each tool answers, so it can tell that a question
      about tabular models belongs here. The entrant pool is exposed as an argument too.
* 2026/08/03-v0.1.7.1:
    * Add new verified model: EXAONE-Tabular (classification only; its regression results are imputed)
* 2026/07/31-v0.1.7:
    * The leaderboard can now be queried by an AI assistant: this Space is an MCP server with
      tools for the TabArena and BeyondArena leaderboards and for the accuracy-versus-time
      Pareto frontier. Setup instructions are under "Agentic Use & Interpretation".
    * Every published leaderboard is also readable as a plain CSV from the Space repo, so
      scripts can use the same numbers the tables show.
    * The full leaderboard table is now interactive: every column sorts, headers explain what
      they measure, and you can filter by model family, model, or variant, then download the
      result as a CSV.
    * New interactive win-rate matrix and Leaderboard Overview, alongside the existing Pareto
      and tuning-trajectory explorers.
    * Every figure opens in paper view (white background, chart and legend only) and can be
      downloaded as SVG, PDF, or PNG. "Edit view" opens the controls, and one click switches
      to the static figure.
    * Figures and subset selectors were restyled so it is clearer which controls belong
      together and where each figure begins.
    * Numbers now use a "." decimal separator whatever your browser locale is, so the tables
      and figures agree.
* 2026/07/21-v0.1.6:
    * Redesigned Pareto-front and tuning-trajectory figures: model-family colors, with the Pareto
      front and highlighted methods in focus while all other methods are greyed out.
    * New interactive Pareto explorers per subset: click methods or model families to highlight
      them, switch the y-axis between Improvability and Elo, and hover any point for exact values;
      partially imputed methods are marked with a dashed ring.
* 2026/07/21-v0.1.5.5:
    * Add new verified model: Nori-30M
* 2026/07/21-v0.1.5.4:
    * Add new unverified model: TabDPT-Turbo
    * Updated existing models with improved train/inference time measurement (untimed environment
      warm-up + persisted inference): CatBoost, ChimeraBoost, EBM, ExtraTrees, Nori, TabFM,
      TabICLv2, TabPFN-3, TabSwift
* 2026/07/10-v0.1.5.3:
    * Add new unverified model: TabSwift
* 2026/07/08-v0.1.5.2:
    * Add new verified model: TabFM
* 2026/06/30-v0.1.5.1:
    * Add new verified models: Nori, ChimeraBoost
* 2026/06/22-v0.1.5:
    * New leaderboard UI: top-level tabs for multiple leaderboards, a cross-subset Elo overview, imputation
      and TabArena-Lite as toggles, an interactive metric reference, and an agentic-use guide.
* 2026/06/02-v0.1.4:
    * Add new verified models: TabPFN-3, iLTM (only 25 configs)
    * Add new unverified model: OrionMSP
    * Finish model integration: LimiX now runs on all datasets.
* 2026/03/25-v0.1.3.1:
    * Add new verified model: TabPFN-2.6
* 2026/03/24-v0.1.3: 
    * Add new verified models: TabICLv2, TabSTAR, PerpetualBooster.
    * Add new verified reference pipeline: AutoGluon 1.5 (extreme, 4h).
    * Added Binary and Multiclass task views.
    * Removed TabPFNv2-data view in results as most recent tabular foundation models work on all tasks on TabArena.
    * Removed AutoGluon 1.4 (extreme, 4h) from results as it is replaced by AutoGluon 1.5 (extreme, 4h).
* 2025/12/11-v0.1.2.2: 
    * Add new unverified model: SAP-RPT-OSS (a.k.a.: ConTextTab, sap-rpt-1-oss)
* 2025/11/27-v0.1.2.1: 
    * Make tuning trajectories start from the default configuration.
    * UI improvements and more user-friendly explanations.
* 2025/11/22-v0.1.2: Add newest version of TabArena LB for NeurIPS 2025
    * New UI and new leaderboard subsets for different dataset sizes, tasks, and imputation + general polish. 
    * Some metrics have been refactored and made more stable (see GitHub for details).
    * Updated Reference Pipeline to include AutoGluon v1.4 with the extreme preset.
    * Updated existing models: RealMLP, TabDPT, EBM
    * Add new verified models: Mitra, xRFM, RealTabPFN-v2.5
    * Add new unverified models: TabFlex, BetaTabPFN, LimiX
* 2025/06/13-v0.1.1: Add data for all subsets and re-runs on GPU; Add leaderboards for subsets;
 new overview; add Figures to LBs.
* 2025/05-v0.1.0: Initialization of the TabArena-v0.1 leaderboard.
Old Leaderboards (with major changes) can be found at:
* [Tabarena-v0.1 and TabArena-v0.1.1](https://huggingface.co/spaces/TabArena-Legacy/TabArena-v0.1.1)
* [Tabarena-v0.1.2.2](https://huggingface.co/spaces/TabArena-Legacy/TabArena-v0.1.2)
* [Tabarena-v0.1.3.1](https://huggingface.co/spaces/TabArena-Legacy/TabArena-v0.1.3)
* [Tabarena-v0.1.9.9](https://huggingface.co/TabArena-Legacy/TabArena-v0.1.9), the last version before the
  redesign (its code and data, to run locally)
"""