Beanbagdzf/scipaths-silver-pipeline

SciPaths silver data pipeline

Private package that builds SciPaths silver pathway annotations from an arXiv paper.

It creates data only. It does not train or score a forecasting agent.

A finished run writes silver_data_pipeline/runs/<job>/silver.json and review.html.

• Stage A fetches the paper and citing papers, labels citation contexts, and writes target and enabling contributions.
• Stage B searches a local BM25 index of arXiv papers (cs.* and stat.ML), then asks Gemini to pick a prior paper or __NONE__.

Layout

Path	Role
silver_data_pipeline/stage_a/step_01 … step_08	Paper fetch through contribution annotation
silver_data_pipeline/stage_b/	BM25 retrieval and Gemini rerank
silver_data_pipeline/Deep-Citation/	Citation-intent classifier (SciBERT). Weights are downloaded by setup.
silver_data_pipeline/scripts/build_bm25_index.py	Build only the local index
silver_data_pipeline/scripts/run_silver_batch.py	Run many papers, resumable
silver_data_pipeline/data/	Local index and CVF id lists. See data/README.md.
silver_data_pipeline/runs/	Run outputs. Not committed.

Prerequisites

Do these before setup. Setup does not create accounts or open a browser. If any of them are missing, it stops and lists what is missing.

1. Python 3.10+ and this repo:

   git clone https://huggingface.co/Beanbagdzf/scipaths-silver-pipeline
   cd scipaths-silver-pipeline
   python -m venv .venv
   source .venv/bin/activate
   pip install -e .
   

   Cloning this private repo requires a Hugging Face login that can read it (huggingface-cli login).

2. API keys. Copy the example file and fill it in:

   cp .env.example silver_data_pipeline/.env
   

   Variable	Where it comes from
   GEMINI_API_KEY	Google AI Studio. Required. Every LLM step uses gemini-3.1-pro-preview.
   SEMANTIC_SCHOLAR_API_KEY	Semantic Scholar API. Required. Without it, citation fetches hit the public rate limit.

   Setup reads silver_data_pipeline/.env only. Do not commit that file.

3. Kaggle, for the arXiv metadata dump that becomes the BM25 index.

   • Create an API token and save it as ~/.kaggle/kaggle.json (or export KAGGLE_USERNAME and KAGGLE_KEY).
   • Open Cornell-University/arxiv and accept the dataset terms while logged into that same account.

   Leave about 10 GB free. Setup keeps a ~5 GB snapshot and a ~4 GB DuckDB file.

4. Hugging Face token if you are not already logged in. The citation checkpoint (best_model.pt, 419 MB) lives in this private repo. huggingface-cli login or HF_TOKEN in the environment is enough. The public base model allenai/scibert_scivocab_uncased is downloaded in the same step and does not need a token.

Setup

From the repo root, with the venv active:

python -m silver_data_pipeline setup

Setup checks the prerequisites above and prints every gap in one message. It does not download anything until those checks pass.

When the checks pass it:

1. Downloads best_model.pt into silver_data_pipeline/Deep-Citation/Workspace/acl_scicite_wksp_trl/ if it is not already there.
2. Prefetches allenai/scibert_scivocab_uncased.
3. Downloads the arXiv metadata snapshot and builds silver_data_pipeline/data/arxiv_bm25.duckdb (cs.* and stat.ML papers, title + abstract). If the DuckDB file already exists, that step is skipped.

Build only the index, after Kaggle is set up:

python silver_data_pipeline/scripts/build_bm25_index.py

That writes under silver_data_pipeline/data/ regardless of the current directory.

If a download fails after the checks, setup exits with the artifact that failed. A Kaggle 403 usually means the dataset terms were not accepted. A Hugging Face 401 means the token cannot read this private repo.

Run one paper

From the repo root:

python -m silver_data_pipeline run --paper https://arxiv.org/abs/2211.08788

Outputs:

• silver_data_pipeline/runs/<job>/silver.json
• silver_data_pipeline/runs/<job>/review.html

PDF input is best-effort (title resolved through Semantic Scholar):

python -m silver_data_pipeline run --pdf /path/to/paper.pdf

Run a batch

CVF id lists are in silver_data_pipeline/data/cvf_s2_mapping/. Example for the citation band 5–10:

python silver_data_pipeline/scripts/run_silver_batch.py \
  --ids silver_data_pipeline/data/cvf_s2_mapping/batch_100_each_cites_5_10.json \
  --output-root silver_data_pipeline/runs/cvf_silver_batch_cites_5_10 \
  --manifest silver_data_pipeline/runs/cvf_silver_batch_cites_5_10/manifest.jsonl

The batch skips papers already marked ok in the manifest. Rebuild the CVF lists with python silver_data_pipeline/scripts/build_cvf_s2_mapping.py (it reads the same .env).

What is not in git

Run folders, .env, the arXiv snapshot, the DuckDB index, and scraped CVF HTML. The citation checkpoint is in this private repo; setup downloads it if the file is not already on disk.