# Local business-record export v1 ## Scope This utility operates only on an **explicitly supplied offline source directory** and a new, disjoint output directory. It is not imported by Frontend or Worker. There are no credentials, network clients, GitHub uploads, timers, operational hooks, deployment changes or automatic restoration. Do not point it at a live Bucket or actual operational data directory. `/data` and descendants are explicitly refused for export, verification and reconstruction; other mounted locations cannot be reliably identified as production, so the caller must supply an authorized **offline copy**. This task used synthetic data only. No dependency changes are necessary; these modules use the Python standard library. ## Added files and functions | File | Responsibility | | --- | --- | | `tools/export_schema.py` | `SCHEMAS`, `PART_SCHEMAS` consumers, v1 typed field definitions; `clean`, `sanitize_text`, `sanitize_url`, fixed-code `ExportError` | | `tools/business_export.py` | `inventory`/`classify`, bounded `source_records`, `Writer.record`/`_chunk`, `validate`, `export`, `verify`, `reconstructed_records`, `reconstruct`, CLI `main` | | `tools/export_fixture.py` | `fixture`/`write` and a local synthetic example CLI, including deliberate fake credential canaries | | `tools/__init__.py` | Package marker only, with no runtime imports or hooks | | `business_export_regression.py` | Focused offline acceptance/failure-boundary tests | | `BUSINESS_EXPORT.md` | This schema, operational limits, results and future-uploader proposal | No existing application or Worker file is modified. The earlier GitHub uploader deferral remains: this task supplies only its local export prerequisite. The schema was checked against `src/submission/{registry,frontend,links,quota,publication}.py`, Worker `src/worker/{runner,lease,quota,provenance,deletions,publications,history_recovery}.py`, storage/history readers and the four task Plugins. Local Git history of `_update_best_if_improved` was also inspected to classify original best-result fields such as `scenes_count`. A regression checks that current `_HISTORY_FIELDS` and all four Plugins' displayed metric names are classified. ## Commands Run from `PhysInOne-Leaderboard` with the local Python environment: ```powershell ..\.venv-frontend\Scripts\python.exe -m tools.export_fixture --output ..\my-synthetic-export ..\.venv-frontend\Scripts\python.exe -m tools.business_export export ` --source ..\my-synthetic-export\synthetic-source ` --output ..\another-export --source-id synthetic-offline-copy ..\.venv-frontend\Scripts\python.exe -m tools.business_export verify --export ..\another-export ..\.venv-frontend\Scripts\python.exe -m tools.business_export reconstruct ` --export ..\another-export --output ..\isolated-reconstruction ..\.venv-frontend\Scripts\python.exe business_export_regression.py ``` Output targets must not exist. Never reuse an operational Queue/results/Registry directory as a reconstruction target. Use one writer per output target and do not change source/export files during verification. Existing output is not merged or overwritten. Exports retain legitimate researcher PII. Output roots request owner-only POSIX directory permissions (`0700`); Windows permissions depend on the parent directory ACL and Python/platform behavior. Use a private local folder and review its ACL before retaining or sharing any authorized real-data export. Do not publish the synthetic-source pattern as a license to export real user data without authorization. The supplied example is under workspace `_offline_export_examples/business-v1-release-20261012/`: - `synthetic-source/`: fake business records **with fake canaries**, not an upload artifact. - `export/manifest.json` and `export/{records,chunks}/`: sanitized export to inspect. - `isolated/business-records.jsonl` and `isolated/verification.json`: reconstructed, isolated representation, not operational restore files. ## Schema and source coverage `schema_version = 1` is attached to manifest/record envelopes, not injected into authoritative business records. The full field-level contract is the explicit `SCHEMAS` dictionary; unknown fields fail the whole export. No generic arbitrary-JSON pass-through is used. ### Identity and quotas `submission_policy/registry.json` and classified `identity_migration_snapshots/<64-hex>.json`: - `submitters`: original storage keys, `hf_username`, `hf_subject`/legacy `hf_sub`, ORCID/researcher/submitter IDs; name, role, affiliation, email and supervisor; identity/ORCID/rules/profile timestamps; original profile, profile revisions and rules acceptances. - Nested profile changes explicitly classify old/new values for the existing profile fields. Revisions and rules acceptances retain order and owner references. - `reservations`: SID, `orcid_id`, `reserved_at`. - `publication_sequences`: original researcher → track → allocated integer counter. - Historical `hf_accounts`: classified username/subject/ID/timestamp fields. It is retained as historical evidence, not treated as new authentication proof. - Consumption/refund timestamps, reasons and events are retained in submissions/History. Rolling quota counts are derived from those records; environment-only quota limits are not present in the supplied source and are not invented or read from Secrets. The Registry is split into a root scaffold plus one record per profile/reservation/account/researcher sequence map, including migration snapshots. There is no raw Registry dump. Migration source filenames identify the originals; integrity hashes refer to **sanitized output**, not to the original filename's raw-data checksum. ### Submission/evaluation - `queue/{pending,processing,complete}/.json`, archived terminal records and archived/live Progress. - IDs, task/track, ownership snapshots, method/display name, source repository/subdirectory, timestamps, primary metric, status, worker/attempt attribution, evaluator configuration version, consumed/refunded quota, cancellation timestamp, score and metrics. - Existing creation snapshot, researcher snapshot, private-token-provided boolean and consent policy/version/acceptance timestamp. - Provenance v1: pinned repository/type/subdirectory/revision/resolution timestamp; evaluator commit/source digest/protocol/manifest identifiers; metric configuration/digest/package versions and requested/effective device. No runtime or model calls occur. - Progress: all results/metrics/timestamps, failures, case start times, retries/attempt counts/outcomes/infrastructure flags, current case/stage/index/attempt, retry timestamps and submission events. - Metric names are explicitly classified for current image metrics, Future Prediction view metrics and Physical Properties' rendering/material parameter metrics. New metrics require a reviewed schema update rather than silent dropping. ### History, presentation, publication and audit - Monthly `_history.YYYY-MM.jsonl`: streamed one row at a time into per-submission record envelopes, preserving source identity and original row ordinal. The monthly source is **not** the canonical growing export blob. - `_deleted_submissions.jsonl`: every tombstone and its owner/task/time/request/event/reason fields. - `presentation_metadata//.json`: display name, current paper/code URLs, update time and ordered revisions including previous/new values and actor/event identity. - `publications.json`: entries, versions/watermarks and historical previous/new publication transitions. - `publication_requests/{pending,processing,complete}` and `queue/deletions/{pending,processing,complete}`: classified actions, IDs, owners/subjects, sequences, replacement target, timestamps/status/message receipts. - `logs/exec-YYYY-MM[.partNN].jsonl`: **classified current business recovery audit events only**. Unknown/mixed legacy execution formats fail; they are not blindly uploaded or silently skipped. - Original `*_best.json`: classified original fields including `team_id`, method, task, primary metric, score, metrics, `scenes_count`, update time and SID; known later owner/deletion fields are classified too. Empty initial best-result placeholders are preserved without fabricating a SID. Events preserve array order and JSONL ordinals. Ordering is validated within each Case/lifecycle group, not by sorting or rewriting the original History; History Recovery can legitimately concatenate lifecycle events and earlier Case events. Case attempts cannot move backwards. No deduplication of source rows/events is performed merely to reduce size. ## Exclusion and sanitization policy The following explicitly classified fields are excluded at **every supported nested object level**: `access_token`, `access_token_encrypted`, `private_token`, `refresh_token`, `token`, `oauth_states`, `orcid_states`, `orcid_verifications`, `oauth_info`, `auth`, `session`, `session_secret`, `session_key`, `cookies`, `cookie`, `authorization`, `password`, `client_secret`, `api_key`, `fernet_key`, `github_token`, `email_api_key`, `SUBMISSION_TOKEN_ENCRYPTION_KEY`, `ORCID_CLIENT_SECRET`, `HF_TOKEN`, `oauth_state`, `verification_code`, `code_verifier`, `csrf_token`. `lease_expires_at` and `evaluator_dispatch_ready` are omitted as live lease/dispatch artifacts. Worker attribution and historical attempt/timestamps remain as business provenance, not a restorable lease. The source scanner never exports GT, prediction archives, extracted media, `/tmp`, raw task/performance/child logs, operational restart state or lock/temp files. A staged cancellation `.submission` record stops collection rather than ignoring an in-progress authoritative transition. Unsupported JSON/JSONL in a scanned business directory stops export rather than being silently omitted. Strings are sanitized for recognizable HF/GitHub/API/JWT token forms, Bearer/Basic authorization, credential assignments (including quoted error/audit text) and credential-bearing URLs. URL userinfo is removed; token/secret/password/signature/access-key/API-key/session/OAuth-code/state-like query or fragment parameters are removed. Ordinary parameters and anchors such as `version=2`, `branch=main` and `#abstract` are retained. Authorization/Cookie header content is redacted. Sanitization is idempotent. Ownership identifiers are **never replaced with anonymous/redacted identifiers**: an identifier containing a recognized credential or an unsupported form refuses export. Legitimate profile PII and ownership mappings remain intact. Unknown nested fields/metrics are refused with a fixed diagnostic code, never raw field names/values or exception content. This is a versioned classification policy, not a universal detector for arbitrary unlabeled secret bytes. Opaque credentials disguised as ordinary prose cannot be reliably inferred from source text alone. Unsupported legacy free-form/authentication formats require review and schema extension; do not treat passing a regex as independent proof that an unreviewed source contains no secrets. Synthetic tests cover nested auth/token fields, URL credentials and error/audit canaries, and scan **all output and reconstruction files** for them. ## Layout, limits and bounded collection ```text export/ manifest.json records/ registry/.json migration/.json publications/.json submissions/// submission-.json history-.json progress-.json metadata/... tombstone/... audit/... chunks/ ...//cases-0000.json ...//submission_events-0000.json ...//profile_revisions-0000.json ``` Record names are deterministic hashes of source-relative identity, ordinal and Registry selector; they do not use content hashes or plaintext credential values. Available task/SID records have readable task/SID directories. Presentation sidecars lacking their own SID/task fields retain their original source-relative reference in a category record. Case chunks group the union of `scene_results`, `scene_failures`, `scene_retries`, `infrastructure_failures` and `case_started_at` by Case, with **at most 200 Cases** per chunk. Large arrays such as events/revisions/publication history are independently chunked in original order, at most 200 items. Chunks are shortened to honor the **5MiB serialized payload limit**. A single oversized Case/item or ordinary record fails; nothing is truncated. Empty/absent/null collections preserve their original distinctions. Each JSON source document/JSONL row is capped at **256MiB (268,435,456 bytes)**, including whitespace and the JSONL newline. JSONL files are streamed, not loaded as whole monthly histories. JSON objects use standard-library parsing of one bounded source record at a time. Input/export payload counts are capped at 100,000 files; the independent manifest size limit remains **32MiB**. Serialized output payloads remain at most **5MiB**, with at most **200 Cases** per chunk. Small files are not read into a preallocated maximum-size buffer. Before parsing a source record, the exporter counts `{`, `[`, `,` and `:` bytes, including occurrences inside strings. More than 250,000 such markers fails with `record_memory_budget_exceeded`. The conservative memory estimate is twelve times serialized bytes plus 512 bytes per marker, capped at 4GiB. Estimated requirements plus a 256MiB reserve must fit currently available physical memory (Windows) or available pages and accessible cgroup limits (Linux). An unavailable capacity probe fails closed with `memory_capacity_unavailable`; insufficient space fails with `insufficient_export_memory`. Punctuation-heavy legitimate records can be conservatively refused; this is not a schema relaxation or truncation. The 256MiB ceiling does not guarantee acceptance on every machine or for every record shape. Verification checks the serialized size of each record and its chunks before materializing the combined object; Registry reconstruction also checks the combined parts. These checks use a conservative 128,000,000-byte structural allowance plus twelve times serialized size, the same 4GiB estimate ceiling and available-memory reserve. Checks are advisory admission estimates, **not an OS-enforced memory limit**; available memory can change and other processes can still cause OOM. Catchable allocation failures produce `export_memory_exhausted` in export/verify/CLI, without publishing a complete target. Run large validation in an isolated process with adequate resources. Hashing canonical logical rows and writing isolated reconstruction use incremental JSON encoding; a single string fragment and standard JSON parsing still allocate whole strings/objects. No constant-memory or universal 256MiB stress-test claim is made. Cross-record indexes retain compact identity/status/aggregate summaries and hashes of snapshots/provenance, not all historical Case/event data. Memory still grows with distinct identities/submissions/manifest entries up to the file-count cap; this is not a constant-RAM database or a promise about OS RSS. Historical compatibility is explicit: Registry `teams` retains the former Team membership schema; `hf_accounts` supports either the object metadata form or the historical username-to-stable-subject string form. String entries are preserved verbatim, never used to create identities, and current Registry aliases must match a unique existing researcher binding. Only these selected Registry parts may have scalar envelope data; ordinary business records still require objects. Older verifier builds that require all envelope data to be objects reject such new exports safely; use the matching updated verifier. `historical-deletion-evidence-v1` permits self-contained standalone tombstones without inventing parents. Every tombstone requires a valid SID, a known task matching its source task, a timezone-qualified deletion timestamp and an owner unambiguously linked to the current Registry. Conflicting duplicates, parent task/owner/time conflicts, matching receipt identity/actor/subject/time contradictions and matching deletion audit contradictions fail. Retained parents cannot be active. A matching completed deletion receipt with corroborated requester identity confirms deletion, **not authorization of physical parent removal**. Without that evidence, valid standalone rows remain `standalone_provenance_incomplete`, never verified. The manifest's recomputed `validation.historical_tombstones` records the policy and unique tombstone counts (`retained_parent`, `standalone_deletion_verified`, `standalone_provenance_incomplete`), explicitly marks parent absence as not restorable, and states that retention authorization is not inferred. Tombstone logical rows and all original fields remain unchanged by this classification. Duplicate identical rows remain present in reconstruction but count once as deletion identities. Only completed deletion receipts may refer to a matching standalone tombstone; Progress, metadata, publications and other non-deletion records still require real parents. There is no ignore-missing-references flag, fabricated score, fabricated terminal state or operational restore. Older validation manifests without the policy metadata fail the new full validation-result comparison rather than being silently upgraded; use matching exporter/verifier versions. See `BUSINESS_EXPORT_TOMBSTONE_POLICY.md` for evidence and Stage A acceptance. Physical Properties metrics include `materials_count`, logarithmic non-Newtonian yield and `_count`/`_std` statistics for a finite, explicitly enumerated property-metric set. Unknown metrics and arbitrary suffixes remain refused. Historical tombstone `team_id` is retained; a legacy username owner can be resolved only through an unambiguous current binding, without rewriting the tombstone. Missing parent submissions still fail. Historical `history_written_at` must be a timezone-qualified timestamp. Operator publication `events` retains a fixed target/old-value/new-value audit schema, with target and submission ownership checks; it does not change publication authority. `queue/worker-heartbeat.json` is live readiness/run health, not durable submission business history. Only the exact four-field non-null schema (`worker_id`, `worker_run_id`, `ready`, `heartbeat_at`) is excluded, after strict type/timestamp validation and a 5MiB read bound. Unexpected fields, malformed JSON or another control-file name still fail; there is no blanket control-directory exclusion. Source immutability auditing of the offline collection includes this excluded file. See `BUSINESS_EXPORT_COMPATIBILITY.md` for production acceptance results and evidence limitations. Collection hashes read source files, compares consumed bytes, rereads hashes and rescans the file inventory. Checks are repeated after validation before publication. At most **three complete collection attempts**, no business locks, no source mutation/repair. This detects practical changed/torn/moved files, not all possible adversarial ABA changes or a global transactional instant. Continuous modification, unreadable files or invalid JSON fail safely; rerun against a settled offline copy. Output is written in a sibling temporary staging directory. Its manifest is written only after schema/relationship validation, then full verification runs before directory rename to the new output target. Failed staging is removed; no complete target is emitted. Export does not overwrite an existing target, and source/output ancestry and symlinks are refused. Do not run concurrent publishers to the same output path or modify the directory tree concurrently. OS crash durability is not a database guarantee; always run `verify` before trusting a snapshot recovered after a crash. ## Validation and manifest Checks include: - Strict valid JSON (duplicate keys/nonfinite numbers and partially written records fail). - Filename/SID/task/status consistency; unique current ORCID/HF bindings; submission/researcher references; profile revision/rules ownership. - Matching terminal/History aggregate scores, metrics, ownership, creation snapshots and provenance; original primary score consistency where recorded. - Progress identity/total/result/failure counts, no Case both done and failed, retry outcome consistency and ordered attempt/event history. - A terminal without its History row, or a terminal with recorded completed/failed Cases but missing Progress, fails rather than pretending all Case/history data is present. The exporter does **not** invoke History Recovery or fabricate missing records. - Provenance input repository/subdirectory/revision relationships and supported schema version. - Tombstones, presentation references, completed request owner/task, current publication's DONE/not-deleted ownership, historical publication references, allocated counters/applied watermarks and matching request identity/subject/sequence. `manifest.json` includes: - `schema_version`, `source_identity`, start/completion UTC timestamps. - `complete=true`, `validation_status=validated_best_effort`, named exclusion policy. - Per-category record counts, total `record_count`, `file_count` (payload files, excluding manifest). - Every relative payload path, serialized byte length, SHA-256, category and record/chunk role. - Source-relative identity/type/row count and SHA-256 of each **full sanitized logical source row**. - Essential validation counts: researchers, unique submissions, Progress records, tombstones and applied publication watermarks. Verification checks hashes/sizes/schema, unique paths and chunk ownership, no unlisted/missing files, source ordinals/counts, cross-record relationships and **reconstructed logical-row hashes**. This last check detects dropped fields/Case records/events even if individual file checksums were recalculated. The manifest is not cryptographically signed: a malicious actor who can rewrite all data and manifest hashes can replace the backup. Private storage/access control and later independent commit authentication remain necessary. ## Reconstruction and limitations `reconstruct` first verifies the export, then reassembles every sanitized source row into `business-records.jsonl` envelopes `{source, ordinal, data}` in an isolated new directory. It restores Registry parts and all Case maps/array chunks exactly, preserving source-row identity/order. `verification.json` contains the row count, validation counts and reconstruction-file checksum. A second verification occurs before publishing the isolated directory. This is **not** an operational filesystem layout, automatic Bucket restoration, identity migration or score recomputation. It preserves recorded scores and metrics without rerunning the models. The source's duplicate representations (terminal/History, migration/current Registry) remain distinct; only their originally required relationships are checked. Soft-delete markers and publication watermarks are retained, not applied to destroy exported records. Known refusal/recovery boundaries: - Legacy owners lacking a unique existing researcher/HF username binding: `unresolved_legacy_owner`; no guessed merges or new accounts. - Unknown Registry/profile/metric/provenance fields, unknown business file formats, mixed legacy exec audit rows, corrupt/partial JSONL tails: stop. No skip-corruption or raw-copy fallback. - Pending transient cancellation staging, missing terminal History/Case records, orphan references or conflicting watermarks: stop and retry/review an offline copy. - Schema-external authentication state, encrypted tokens, live leases and runtime dispatch flags are intentionally unrecoverable. Private unfinished jobs require new credentials. - Original legacy aggregate-only best records and empty placeholders have no invented Case metrics/terminal status. Their recorded fields are preserved; no complete per-Case history is asserted for data the legacy record never contained. - No raw logs/media/GT/models/environment-only parameters or secrets are backed up. Current schemas can be validated without checking inaccessible external repositories or verifying genuine ORCID/HF authentication again. - `complete` covers the classified, validated records in the explicitly supplied source copy. It cannot prove that the copy includes every file ever present in production or that an absent optional directory was never lost earlier. Any later production restore must be a separately authorized administrative operation in an isolated test environment first, respecting tombstones, publication sequences and ownership. Never blindly restore runtime leases or reuse old authentication secrets. ## Results and resource measurements The delivered small example contains one researcher, eight unique submissions (four tasks plus queued/running/cancelled/legacy records), four Progress records, one deletion tombstone and one publication watermark. Synthetic source rows are reconstructed and compared to their explicit sanitized schemas and logical-row checksums; source content and modification times remain unchanged. Measurement methodology: local Python 3.12; one selected task's Case count below plus three Cases each in the other tasks; Registry/migration/active/legacy/publication/audit fixtures also included. Fixture creation precedes `tracemalloc`. Numbers cover export, its verification and isolated reconstruction, not network/GPU work; traced Python allocations are **not process RSS** or production upper bounds. Cases have simple synthetic metrics and two events plus one recovered retry. | Selected task | Cases | Total export bytes (including manifest) | Payload files | Largest payload | Traced peak bytes | Export + verify + reconstruction | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | Physical Properties | 60 | 177,862 | 62 | 38,081 | 1,367,880 | 2.185s | | Future Prediction | 103 | 244,298 | 64 | 63,203 | 1,405,331 | 2.787s | | Motion Transfer | 214 | 414,015 | 67 | 63,203 | 2,148,401 | 4.692s | | Video Generation | 200 | 391,934 | 65 | 63,203 | 2,013,965 | 4.565s | | Motion Transfer moderate stress fixture | 2,140 | 3,367,844 | 114 | 63,203 | 18,436,865 | 31.017s | All five measurements passed checksum, relationship and reconstruction verification. Timing includes `tracemalloc` instrumentation and local filesystem overhead; it is not a GPU/online estimate. These synthetic fixtures have many small records, not one file/request per Case. The utility was not optimized for the 15,173-Case full Video Generation manifest, nor run on production data. Offline regression results: - `business_export_regression.py`: **26/26 passed**, covering the requested categories, known nested fields and canaries, legacy/migration/empty placeholders, small/moderate Cases, exact sanitized-row reconstruction, deterministic repeat exports, source hash/mtime preservation, incomplete JSON, unknown fields, missing/conflicting relationships, bounded source-change retry, 200-Case/adaptive byte boundaries, oversized single Case, manifest/path/tamper checks and missing terminal History/Progress. - Existing `identity_history_regression.py`: **21/21 passed**. - Existing `stable_identity_regression.py`: **27/27 passed**. - Existing `lifecycle_regression.py`: **49/49 passed**. - Existing `publication_order_regression.py`: **25/25 passed**. - Python compilation and whitespace/diff checks passed. No actual application integration, GPU, production data, GitHub authentication, email call, deployment, commit or push occurred. The release synthetic example passed CLI verification and reconstructed **28 source rows**, preserving the stated one researcher/eight submissions/four Progress records/one tombstone/one watermark. Reconstruction file SHA-256: `f2e595ef277686e2a90c1e8ae4f2722da668f779162e14559da3fcbc5a6b90d2` Use the actual example manifest for each payload checksum; copying a checksum from this document is not a substitute for running `verify`. ## Future uploader proposal (not implemented) Only consume an export that passes `verify`. Check a dedicated destination repository is private; keep a repository-scoped credential outside exported records. Compare deterministic payload hashes with the previous manifest and upload changed files plus a complete new manifest/trigger receipt. Serialize uploads separately, use a non-forced branch update, and on remote conflicts recollect rather than overwriting newer data. Confirm the remote version before local success; recover uncertain outcomes from a per-terminal receipt. A later separately reviewed trigger can baseline existing terminal records and discover missed Complete/Archive events. This module provides **none** of that mechanism and makes no live GitHub/email/deployment claims. No source changes or operational integration are needed to use this local utility.