HF dataset-viewer fix: configs + metadata.jsonl on share/compress, auto-repair sweep at startup
Browse filesImplements docs/hf-viewer-fix-spec.md (root cause: extension-inferred
JSON loader chokes on motion files and disables the whole viewer).
Share & compress paths (deliverable A):
- viewerConfigPlan/renderViewerConfigBlock/ensureViewerConfig/
buildViewerMetadataRows/buildMetadataJsonl/parseMetadataJsonl helpers
(exported for tests); ensureViewerConfig only touches the frontmatter's
configs: key — byte-preserving, idempotent
- uploadMove: README is no longer regenerated on every share (the old
clobber-customized-READMEs failure mode). Fresh repos get the community
README (+configs when audio); existing repos get only the configs key
ensured; metadata.jsonl rebuilt to cover ALL audio in the repo
(titles = stems, descriptions from local moves via opts.localMoves,
existing rows kept). Rename-cleanup-doomed files excluded (rule 4).
fetchRepoText distinguishes 404 from network failure: on failure the
share proceeds WITHOUT touching README/metadata (never clobber).
- compressAndRepublish: README(ensured) + metadata.jsonl ride the same
commit, computed against post-compress state (.ogg in, stale .wav out)
- folder.js passes the folder's moves as the metadata description source
Startup auto-repair (deliverable B):
- HF.repairOwnedViewerConfigs: own-namespace only, skip READMEs already
carrying configs, skip no-audio repos, one commit per repo, per-repo
try/catch, {checked,fixed,errors}
- main.js runViewerConfigMigration: runs automatically post-connect,
gated by IndexedDB kv 'viewer_config_migration_v1' so it's once per
browser profile — flag only written on a zero-error sweep (flaky runs
retry next session); bump the key to force a global re-sweep
- listSharedMoves projection gains description/sharedStem/fromCommunity
Compatibility: no move/audio file renamed or rewritten anywhere; import
pairing and countMoves ignore metadata.jsonl (endsWith '.json' filters).
35 new tests (viewer helpers, uploadMove viewer paths, sweep guards);
96/96 green.
- docs/hf-viewer-fix-spec.md +169 -0
- lib/db.js +6 -0
- lib/hf.js +386 -25
- main.js +59 -1
- tests/unit/hf-upload.test.js +112 -0
- tests/unit/hf-viewer-sweep.test.js +116 -0
- tests/unit/viewer-config.test.js +184 -0
- views/folder.js +4 -0
|
@@ -0,0 +1,169 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Spec: make shared datasets playable in the HF dataset-viewer
|
| 2 |
+
|
| 3 |
+
**Goal:** every dataset Marionette publishes (new shares AND already-published ones
|
| 4 |
+
owned by the signed-in user) gets a working Hugging Face dataset-viewer page with a
|
| 5 |
+
playable audio column — without changing the on-Hub data layout in any way.
|
| 6 |
+
|
| 7 |
+
Status: root cause diagnosed and fix **verified live** on 2026-07-03. Two repos were
|
| 8 |
+
fixed manually as proof:
|
| 9 |
+
|
| 10 |
+
- Test bed: `RemiFabre/marionette-viewer-test` (viewer works; safe to delete)
|
| 11 |
+
- Production: `pollen-robotics/reachy-mini-emotions-library` (81 rows, playable
|
| 12 |
+
audio + title + description columns). Look at its `README.md` and
|
| 13 |
+
`metadata.jsonl` on the Hub — they are the reference output of this spec.
|
| 14 |
+
|
| 15 |
+
This spec describes app-side work in `lib/hf.js` / `main.js` so the fix applies
|
| 16 |
+
automatically. Nothing here changes the move file format.
|
| 17 |
+
|
| 18 |
+
---
|
| 19 |
+
|
| 20 |
+
## 1. Root cause (why the viewer is broken today)
|
| 21 |
+
|
| 22 |
+
The viewer picks a loader by **file-extension inference** when the README has no
|
| 23 |
+
`configs:` YAML. Our repos contain `<stem>.json` motion files, so it picks the
|
| 24 |
+
JSON builder, which chokes on the motion format:
|
| 25 |
+
|
| 26 |
+
```
|
| 27 |
+
pyarrow.lib.ArrowInvalid: JSON parse error: Column() changed from object to number
|
| 28 |
+
```
|
| 29 |
+
|
| 30 |
+
One failed config disables the **entire** viewer, audio previews included
|
| 31 |
+
(`is-valid` returns all-false, e.g. `Anne-Charlotte/reachy-songs`).
|
| 32 |
+
|
| 33 |
+
## 2. The fix (what a "viewer-ready" repo looks like)
|
| 34 |
+
|
| 35 |
+
Two additive files; zero changes to motion/audio files:
|
| 36 |
+
|
| 37 |
+
**(a) README frontmatter gains a `configs:` key** pointing the viewer at the audio
|
| 38 |
+
files only. This forces the `audiofolder` builder and the motion JSONs are ignored:
|
| 39 |
+
|
| 40 |
+
```yaml
|
| 41 |
+
configs:
|
| 42 |
+
- config_name: default
|
| 43 |
+
data_files:
|
| 44 |
+
- split: train
|
| 45 |
+
path:
|
| 46 |
+
- "data/*.ogg" # adjust to actual layout + extensions, see rules
|
| 47 |
+
- "data/metadata.jsonl"
|
| 48 |
+
```
|
| 49 |
+
|
| 50 |
+
**(b) `metadata.jsonl`** (optional but required by this spec — without it the
|
| 51 |
+
viewer shows anonymous audio rows). One JSON line per audio file, living **in the
|
| 52 |
+
same directory as the audio**, `file_name` relative to that directory:
|
| 53 |
+
|
| 54 |
+
```jsonl
|
| 55 |
+
{"file_name": "amazed1.ogg", "title": "amazed1", "description": "When you discover something extraordinary…", "motion_file": "amazed1.json"}
|
| 56 |
+
```
|
| 57 |
+
|
| 58 |
+
Columns: `title` = move stem, `description` = the move JSON's `description` field,
|
| 59 |
+
`motion_file` = the sidecar's motion file. Do **not** name a column `label`
|
| 60 |
+
(reserved by audiofolder for class labels).
|
| 61 |
+
|
| 62 |
+
### Hard rules (each one was hit or verified during the manual fix)
|
| 63 |
+
|
| 64 |
+
1. **Every glob listed in `data_files` must match ≥ 1 file**, or the config errors
|
| 65 |
+
and the viewer stays broken. Therefore: emit one glob per audio extension
|
| 66 |
+
actually present in the repo (`.ogg` / `.opus` / `.wav`), and **omit the whole
|
| 67 |
+
`configs:` block when the repo has no audio at all**.
|
| 68 |
+
2. Globs are relative to the repo root. Our repos come in two layouts: files under
|
| 69 |
+
`data/` (JS-app uploads) or at the repo root (e.g. the Python-seeded
|
| 70 |
+
emotions/dances libraries). Generate globs for the layout the repo actually has.
|
| 71 |
+
3. The metadata file must be named **exactly `metadata.jsonl`** — NEVER
|
| 72 |
+
`metadata.json`: a `.json` name would be ingested as a move by BOTH loaders
|
| 73 |
+
(`reachy_mini` globs `*.json`; our importer filters `endsWith(".json")`).
|
| 74 |
+
`.jsonl` is provably invisible to both (verified against their source).
|
| 75 |
+
4. Every `file_name` in `metadata.jsonl` must reference an audio file that exists
|
| 76 |
+
in the repo **after the commit** (mind `compressAndRepublish` deleting stale
|
| 77 |
+
`.wav` files in the same operation).
|
| 78 |
+
5. Preserve everything already in the README — especially the
|
| 79 |
+
`reachy_mini_community_moves` tag (Community-list discovery depends on it) and
|
| 80 |
+
any hand-written body (the emotions library README is hand-maintained).
|
| 81 |
+
6. `marionette.json` must never appear in `data_files`.
|
| 82 |
+
|
| 83 |
+
## 3. Deliverable A — new & updated shares emit the fix
|
| 84 |
+
|
| 85 |
+
Touch points in `lib/hf.js`:
|
| 86 |
+
|
| 87 |
+
- **`buildCommunityReadme()` (~line 602)** — for fresh repos, embed the `configs:`
|
| 88 |
+
block in the generated frontmatter. It now needs to know which audio extensions
|
| 89 |
+
the repo will contain (pass this in from callers). No audio → no block.
|
| 90 |
+
- **New helper `ensureViewerConfig(readmeText, globs)`** — for repos that already
|
| 91 |
+
have a README: parse the frontmatter, insert or update **only** the `configs:`
|
| 92 |
+
key, and leave every other byte untouched. This is mandatory for the repair
|
| 93 |
+
sweep (Deliverable B) and recommended for `uploadMove` on existing repos —
|
| 94 |
+
full README regeneration would clobber customized READMEs like the emotions
|
| 95 |
+
library's (which already carries this fix; regenerating it would silently
|
| 96 |
+
remove the block — that is the current-code failure mode this spec eliminates).
|
| 97 |
+
- **`uploadMove()` (~line 381)** — in the same commit as the move, write/refresh
|
| 98 |
+
`data/metadata.jsonl` covering **all** moves in the repo, not just the new one.
|
| 99 |
+
Source for titles/descriptions: the local folder's moves in IndexedDB (labels,
|
| 100 |
+
descriptions, stems). Mind stem rules: local shares use `slugify(move.label)`,
|
| 101 |
+
imported moves keep their upstream stem (see the `m.id.split("/").pop()` logic
|
| 102 |
+
in `compressAndRepublish`, ~line 487). For repo files with no local
|
| 103 |
+
counterpart, keep their existing metadata row if present, else emit a row with
|
| 104 |
+
`title` = stem and no description.
|
| 105 |
+
- **`compressAndRepublish()` (~line 475)** — regenerate README (via
|
| 106 |
+
`ensureViewerConfig`) and `metadata.jsonl` inside its single commit. Audio is
|
| 107 |
+
forced to Opus here, so post-commit extensions are `.ogg`/`.opus` — rule 4.
|
| 108 |
+
- `publishFolderTuning()` — no change needed.
|
| 109 |
+
|
| 110 |
+
## 4. Deliverable B — startup auto-repair of already-published datasets
|
| 111 |
+
|
| 112 |
+
Copy the existing, proven pattern: `HF.repairOwnedCommunityTags()`
|
| 113 |
+
(`lib/hf.js:325-379`) driven by `runCommunityTagMigration()` (`main.js:437-453`) —
|
| 114 |
+
once per session when a token is available, dedup shared moves to unique repos,
|
| 115 |
+
skip repos not in the user's namespace, per-repo try/catch, `{checked, fixed,
|
| 116 |
+
errors}` telemetry, success toast.
|
| 117 |
+
|
| 118 |
+
New sweep (parallel function, or extend the existing one):
|
| 119 |
+
|
| 120 |
+
For each unique owned repo behind locally-shared moves:
|
| 121 |
+
|
| 122 |
+
1. Fetch `README.md` (`resolve/main/README.md`). If its frontmatter already has a
|
| 123 |
+
`configs:` key → **skip** (idempotent; never clobber).
|
| 124 |
+
2. List the repo files (`listDatasetFiles`, ~line 88). No audio files → skip.
|
| 125 |
+
3. Build the extension-accurate globs + `metadata.jsonl` (rules above; local
|
| 126 |
+
IndexedDB is the description source; fall back to omitting descriptions for
|
| 127 |
+
moves not known locally).
|
| 128 |
+
4. Push `README.md` (via `ensureViewerConfig`) + `metadata.jsonl` in **one**
|
| 129 |
+
commit, e.g. `"Marionette: enable dataset-viewer audio previews"`.
|
| 130 |
+
|
| 131 |
+
Scope note: keep the strict own-namespace check. Org repos (e.g.
|
| 132 |
+
`pollen-robotics/*`) are out of scope for the sweep — the emotions library is
|
| 133 |
+
already fixed manually; `pollen-robotics/reachy-mini-dances-library` still has
|
| 134 |
+
the broken viewer and can get the same manual one-commit treatment.
|
| 135 |
+
|
| 136 |
+
## 5. Compatibility invariants (the "0% breakage" contract)
|
| 137 |
+
|
| 138 |
+
- On-Hub data layout unchanged: no move `.json`/audio file is renamed, moved, or
|
| 139 |
+
rewritten by this feature. Only `README.md` + `metadata.jsonl` are written.
|
| 140 |
+
- `reachy_mini` (`src/reachy_mini/motion/recorded_move.py`, GitHub
|
| 141 |
+
`pollen-robotics/reachy_mini`): `RecordedMoves.process()` globs `*.json` and
|
| 142 |
+
`data/*.json`, pairs audio by stem — sees no difference.
|
| 143 |
+
- Marionette JS importer (`importDataset`, `lib/hf.js:121-276`): ingests
|
| 144 |
+
`endsWith(".json")` except `marionette.json`, pairs audio by stem — sees no
|
| 145 |
+
difference. Same for `countMoves`.
|
| 146 |
+
- Python Marionette app publishes the same format; the YAML `configs:` key is
|
| 147 |
+
inert metadata to it. (If it also regenerates READMEs on share, it will need
|
| 148 |
+
the same `ensureViewerConfig` treatment eventually — out of scope here, but
|
| 149 |
+
don't make the JS format diverge.)
|
| 150 |
+
|
| 151 |
+
## 6. Verification (required before calling this done)
|
| 152 |
+
|
| 153 |
+
1. Unit tests: `ensureViewerConfig` is idempotent, preserves an arbitrary README
|
| 154 |
+
body byte-for-byte outside the `configs:` key, handles no-audio (emits
|
| 155 |
+
nothing / removes nothing), multi-extension repos, root vs `data/` layouts.
|
| 156 |
+
2. Live end-to-end on a scratch dataset under the user's namespace (pattern:
|
| 157 |
+
`RemiFabre/marionette-viewer-test`): share a move from the app, then poll —
|
| 158 |
+
|
| 159 |
+
```bash
|
| 160 |
+
curl -s "https://datasets-server.huggingface.co/is-valid?dataset=<repo>" # want viewer:true (takes 1–4 min)
|
| 161 |
+
curl -s "https://datasets-server.huggingface.co/splits?dataset=<repo>" # want failed: []
|
| 162 |
+
curl -s "https://datasets-server.huggingface.co/first-rows?dataset=<repo>&config=default&split=train"
|
| 163 |
+
# want features [audio:Audio, title, description, motion_file] and playable row assets
|
| 164 |
+
```
|
| 165 |
+
|
| 166 |
+
3. Re-import the scratch dataset through the app's Community flow: move count and
|
| 167 |
+
audio pairing identical to pre-change.
|
| 168 |
+
4. For the repair path, byte-diff the README before/after on a repo with a
|
| 169 |
+
customized body: the only difference must be the `configs:` block.
|
|
@@ -137,6 +137,12 @@ export async function listSharedMoves() {
|
|
| 137 |
label: m.label,
|
| 138 |
shared: m.shared,
|
| 139 |
sharedUrl: m.sharedUrl,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 140 |
}));
|
| 141 |
}
|
| 142 |
|
|
|
|
| 137 |
label: m.label,
|
| 138 |
shared: m.shared,
|
| 139 |
sharedUrl: m.sharedUrl,
|
| 140 |
+
// Used by the viewer-config repair sweep: description feeds the
|
| 141 |
+
// metadata.jsonl rows, sharedStem/fromCommunity resolve which
|
| 142 |
+
// on-Hub stem the move publishes under. Still no blobs.
|
| 143 |
+
description: m.description,
|
| 144 |
+
sharedStem: m.sharedStem,
|
| 145 |
+
fromCommunity: m.fromCommunity,
|
| 146 |
}));
|
| 147 |
}
|
| 148 |
|
|
@@ -109,6 +109,21 @@ export const HF = {
|
|
| 109 |
return res;
|
| 110 |
},
|
| 111 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 112 |
async countMoves(repoId, token) {
|
| 113 |
// Cheap-ish probe: list files and count .json entries. Used by the
|
| 114 |
// community list so each row can show the real number before import.
|
|
@@ -378,6 +393,71 @@ export const HF = {
|
|
| 378 |
return { checked, fixed, errors };
|
| 379 |
},
|
| 380 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 381 |
async uploadMove(move, folderName, token, opts = {}) {
|
| 382 |
if (!token) throw new Error("Sign in to Hugging Face to share.");
|
| 383 |
const username = await this.getHfNamespace(token);
|
|
@@ -428,17 +508,6 @@ export const HF = {
|
|
| 428 |
path: `data/${moveStem}.json`,
|
| 429 |
content: new Blob([JSON.stringify(moveJson, null, 2)], { type: "application/json" }),
|
| 430 |
},
|
| 431 |
-
// The community-tag README is REQUIRED for the dataset to
|
| 432 |
-
// appear in the Marionette "Community" tab. Without this
|
| 433 |
-
// file, HF doesn't carry the `reachy_mini_community_moves`
|
| 434 |
-
// tag, and our list endpoint (?filter=…) excludes the
|
| 435 |
-
// dataset. Existing Python Marionette uses the same tag
|
| 436 |
-
// for cross-app discovery. Re-uploaded on every share to
|
| 437 |
-
// keep the frontmatter current if it ever drifts.
|
| 438 |
-
{
|
| 439 |
-
path: "README.md",
|
| 440 |
-
content: new Blob([buildCommunityReadme(folderName, repoId)], { type: "text/markdown" }),
|
| 441 |
-
},
|
| 442 |
];
|
| 443 |
if (move.audioBlob) {
|
| 444 |
// Name the sidecar by its real container so the daemon's loader
|
|
@@ -457,6 +526,81 @@ export const HF = {
|
|
| 457 |
});
|
| 458 |
}
|
| 459 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 460 |
await hfUploadFiles({
|
| 461 |
repo: { type: "dataset", name: repoId },
|
| 462 |
credentials,
|
|
@@ -470,9 +614,8 @@ export const HF = {
|
|
| 470 |
// files still carry the old slug; drop them so community import
|
| 471 |
// doesn't show the move twice. A failed delete only leaves an orphan
|
| 472 |
// (a harmless duplicate), never data loss.
|
| 473 |
-
const oldStem = move.sharedStem;
|
| 474 |
if (oldStem && oldStem !== moveStem) {
|
| 475 |
-
if (
|
| 476 |
// Slugs are slugify(label) with no uniqueness: another move
|
| 477 |
// in this folder currently publishes under the old slug, so
|
| 478 |
// `data/<oldStem>.*` is that move's LIVE copy — deleting it
|
|
@@ -565,21 +708,64 @@ export const HF = {
|
|
| 565 |
m.frameCount = moveJson.set_target_data.length;
|
| 566 |
await putMove(m);
|
| 567 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 568 |
await hfUploadFiles({
|
| 569 |
repo: { type: "dataset", name: repoId },
|
| 570 |
credentials,
|
| 571 |
files,
|
| 572 |
commitTitle: "Compress: 50 Hz motion + Opus audio",
|
| 573 |
});
|
| 574 |
-
// Delete only the .wav sidecars we've just replaced with .ogg — never
|
| 575 |
-
// a .wav whose move we didn't re-upload (that would orphan its audio).
|
| 576 |
-
const upstream = await this.listDatasetFiles(repoId, token);
|
| 577 |
-
const staleWavs = upstream
|
| 578 |
-
.map((f) => f.path)
|
| 579 |
-
.filter((p) => {
|
| 580 |
-
const mm = p.match(/^(?:data\/)?(.+)\.wav$/i);
|
| 581 |
-
return mm && audioStems.has(mm[1]);
|
| 582 |
-
});
|
| 583 |
if (staleWavs.length) {
|
| 584 |
await hfDeleteFiles({
|
| 585 |
repo: { type: "dataset", name: repoId },
|
|
@@ -630,6 +816,178 @@ export const HF = {
|
|
| 630 |
},
|
| 631 |
};
|
| 632 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 633 |
// Build the optional `marionette.json` payload, OR return null when
|
| 634 |
// the folder has no non-default settings worth recording. The schema
|
| 635 |
// version exists so we can add fields later without ambiguity.
|
|
@@ -645,9 +1003,12 @@ function buildMarionetteMeta({ audioOffsetMs } = {}) {
|
|
| 645 |
// README that carries the community discovery tag in YAML
|
| 646 |
// frontmatter. Mirrors the format the Python Marionette app uses, so
|
| 647 |
// datasets published from either app are indistinguishable to the
|
| 648 |
-
// listing endpoint.
|
| 649 |
-
|
|
|
|
|
|
|
| 650 |
const pretty = (folderName || "marionette").replace(/[-_]+/g, " ");
|
|
|
|
| 651 |
return `---
|
| 652 |
license: apache-2.0
|
| 653 |
task_categories:
|
|
@@ -657,7 +1018,7 @@ language:
|
|
| 657 |
tags:
|
| 658 |
- ${HF_COMMUNITY_TAG}
|
| 659 |
pretty_name: ${pretty} • Reachy Mini Moves
|
| 660 |
-
---
|
| 661 |
|
| 662 |
# ${pretty} • Reachy Mini Moves
|
| 663 |
|
|
|
|
| 109 |
return res;
|
| 110 |
},
|
| 111 |
|
| 112 |
+
// Fetch a small text file from a repo, distinguishing "definitely
|
| 113 |
+
// absent" from "couldn't tell". Returns the text, or null on a clean
|
| 114 |
+
// 404, and THROWS on anything else (network flake, 5xx) — callers
|
| 115 |
+
// that decide "no README → generate a fresh one" must not mistake a
|
| 116 |
+
// transient failure for absence and clobber a customized file.
|
| 117 |
+
async fetchRepoText(repoId, path, token) {
|
| 118 |
+
try {
|
| 119 |
+
const res = await this.downloadFile(repoId, path, token);
|
| 120 |
+
return await res.text();
|
| 121 |
+
} catch (e) {
|
| 122 |
+
if (/HF 404/.test(String(e?.message || e))) return null;
|
| 123 |
+
throw e;
|
| 124 |
+
}
|
| 125 |
+
},
|
| 126 |
+
|
| 127 |
async countMoves(repoId, token) {
|
| 128 |
// Cheap-ish probe: list files and count .json entries. Used by the
|
| 129 |
// community list so each row can show the real number before import.
|
|
|
|
| 393 |
return { checked, fixed, errors };
|
| 394 |
},
|
| 395 |
|
| 396 |
+
// Startup auto-repair (docs/hf-viewer-fix-spec.md, deliverable B):
|
| 397 |
+
// make every dataset the user owns viewer-ready. For each unique
|
| 398 |
+
// owned repo behind their locally-shared moves / linked folders:
|
| 399 |
+
// skip if the README already has a `configs:` key (idempotent, never
|
| 400 |
+
// clobber), skip if the repo has no audio, otherwise push README
|
| 401 |
+
// (via ensureViewerConfig — byte-preserving) + a rebuilt
|
| 402 |
+
// metadata.jsonl in ONE commit. Same shape as
|
| 403 |
+
// repairOwnedCommunityTags: strict own-namespace check, per-repo
|
| 404 |
+
// try/catch, { checked, fixed, errors } telemetry.
|
| 405 |
+
//
|
| 406 |
+
// `repos`: Map(repoId → localMoves[]) — the local IndexedDB moves
|
| 407 |
+
// known to live in each repo, used as the title/description source.
|
| 408 |
+
async repairOwnedViewerConfigs({ token, repos, onProgress }) {
|
| 409 |
+
if (!token || !repos?.size) return { checked: 0, fixed: 0, errors: 0 };
|
| 410 |
+
|
| 411 |
+
let myNamespace;
|
| 412 |
+
try {
|
| 413 |
+
myNamespace = await this.getHfNamespace(token);
|
| 414 |
+
} catch (e) {
|
| 415 |
+
Log.warn("repairOwnedViewerConfigs: namespace lookup failed", { error: String(e) });
|
| 416 |
+
return { checked: 0, fixed: 0, errors: 1 };
|
| 417 |
+
}
|
| 418 |
+
|
| 419 |
+
const credentials = { accessToken: token };
|
| 420 |
+
let checked = 0, fixed = 0, errors = 0;
|
| 421 |
+
for (const [repoId, localMoves] of repos) {
|
| 422 |
+
if (repoId.split("/")[0] !== myNamespace) continue; // not ours
|
| 423 |
+
checked++;
|
| 424 |
+
try {
|
| 425 |
+
const existingReadme = await this.fetchRepoText(repoId, "README.md", token);
|
| 426 |
+
if (existingReadme !== null && hasViewerConfig(existingReadme)) continue; // already fixed
|
| 427 |
+
|
| 428 |
+
const paths = (await this.listDatasetFiles(repoId, token)).map((f) => f.path);
|
| 429 |
+
const plan = viewerConfigPlan(paths);
|
| 430 |
+
if (!plan) continue; // no audio → a config would only break the viewer
|
| 431 |
+
|
| 432 |
+
const readme = existingReadme === null
|
| 433 |
+
? buildCommunityReadme(repoId.split("/").pop(), repoId, plan.dataFiles)
|
| 434 |
+
: ensureViewerConfig(existingReadme, plan.dataFiles);
|
| 435 |
+
const existingRows = parseMetadataJsonl(
|
| 436 |
+
paths.includes(plan.metadataPath)
|
| 437 |
+
? await this.fetchRepoText(repoId, plan.metadataPath, token)
|
| 438 |
+
: null,
|
| 439 |
+
);
|
| 440 |
+
const rows = buildViewerMetadataRows({ paths, plan, localMoves, existingRows });
|
| 441 |
+
|
| 442 |
+
await hfUploadFiles({
|
| 443 |
+
repo: { type: "dataset", name: repoId },
|
| 444 |
+
credentials,
|
| 445 |
+
files: [
|
| 446 |
+
{ path: "README.md", content: new Blob([readme], { type: "text/markdown" }) },
|
| 447 |
+
{ path: plan.metadataPath, content: new Blob([buildMetadataJsonl(rows)], { type: "application/json" }) },
|
| 448 |
+
],
|
| 449 |
+
commitTitle: "Marionette: enable dataset-viewer audio previews",
|
| 450 |
+
});
|
| 451 |
+
fixed++;
|
| 452 |
+
onProgress?.(fixed, repos.size);
|
| 453 |
+
} catch (e) {
|
| 454 |
+
errors++;
|
| 455 |
+
Log.warn("repairOwnedViewerConfigs: per-repo failure", { repoId, error: String(e) });
|
| 456 |
+
}
|
| 457 |
+
}
|
| 458 |
+
return { checked, fixed, errors };
|
| 459 |
+
},
|
| 460 |
+
|
| 461 |
async uploadMove(move, folderName, token, opts = {}) {
|
| 462 |
if (!token) throw new Error("Sign in to Hugging Face to share.");
|
| 463 |
const username = await this.getHfNamespace(token);
|
|
|
|
| 508 |
path: `data/${moveStem}.json`,
|
| 509 |
content: new Blob([JSON.stringify(moveJson, null, 2)], { type: "application/json" }),
|
| 510 |
},
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 511 |
];
|
| 512 |
if (move.audioBlob) {
|
| 513 |
// Name the sidecar by its real container so the daemon's loader
|
|
|
|
| 526 |
});
|
| 527 |
}
|
| 528 |
|
| 529 |
+
// README + dataset-viewer support, in the SAME commit as the move
|
| 530 |
+
// (docs/hf-viewer-fix-spec.md):
|
| 531 |
+
// - fresh repos get the generated community README — it carries
|
| 532 |
+
// the reachy_mini_community_moves tag the Community tab's
|
| 533 |
+
// listing filter depends on, plus the viewer `configs:` block
|
| 534 |
+
// when the repo will contain audio;
|
| 535 |
+
// - existing repos get ONLY their `configs:` key inserted or
|
| 536 |
+
// updated (ensureViewerConfig) — never a wholesale rewrite,
|
| 537 |
+
// which used to clobber customized READMEs;
|
| 538 |
+
// - metadata.jsonl is refreshed to cover ALL audio in the repo
|
| 539 |
+
// (titles from stems, descriptions from local IndexedDB moves
|
| 540 |
+
// via opts.localMoves, existing rows kept for files we don't
|
| 541 |
+
// know locally).
|
| 542 |
+
// Any failure here is logged and skipped — viewer housekeeping
|
| 543 |
+
// must never block a share; the startup repair sweep finishes the
|
| 544 |
+
// job next session.
|
| 545 |
+
const oldStem = move.sharedStem;
|
| 546 |
+
const cleanupPlanned = !!(oldStem && oldStem !== moveStem
|
| 547 |
+
&& !(opts.protectedStems || []).includes(oldStem));
|
| 548 |
+
try {
|
| 549 |
+
const upstream = await this.listDatasetFiles(repoId, token);
|
| 550 |
+
// Predicted post-share repo content: what's upstream, minus the
|
| 551 |
+
// old-stem files the rename cleanup below is about to delete,
|
| 552 |
+
// plus everything in this commit (rule 4: metadata/globs must
|
| 553 |
+
// only reference files that exist once the dust settles).
|
| 554 |
+
const staleNames = cleanupPlanned
|
| 555 |
+
? new Set([`${oldStem}.json`, ...SOUND_EXTENSIONS.map((ext) => `${oldStem}${ext}`)])
|
| 556 |
+
: new Set();
|
| 557 |
+
const predicted = new Set(
|
| 558 |
+
upstream.map((f) => f.path).filter((p) => !staleNames.has(p.split("/").pop())),
|
| 559 |
+
);
|
| 560 |
+
for (const f of files) predicted.add(f.path);
|
| 561 |
+
const plan = viewerConfigPlan([...predicted]);
|
| 562 |
+
|
| 563 |
+
const existingReadme = await this.fetchRepoText(repoId, "README.md", token);
|
| 564 |
+
if (existingReadme === null) {
|
| 565 |
+
files.push({
|
| 566 |
+
path: "README.md",
|
| 567 |
+
content: new Blob([buildCommunityReadme(folderName, repoId, plan?.dataFiles)], { type: "text/markdown" }),
|
| 568 |
+
});
|
| 569 |
+
} else if (plan) {
|
| 570 |
+
const ensured = ensureViewerConfig(existingReadme, plan.dataFiles);
|
| 571 |
+
if (ensured !== existingReadme) {
|
| 572 |
+
files.push({ path: "README.md", content: new Blob([ensured], { type: "text/markdown" }) });
|
| 573 |
+
}
|
| 574 |
+
}
|
| 575 |
+
|
| 576 |
+
if (plan) {
|
| 577 |
+
const existingRows = parseMetadataJsonl(
|
| 578 |
+
predicted.has(plan.metadataPath)
|
| 579 |
+
? await this.fetchRepoText(repoId, plan.metadataPath, token)
|
| 580 |
+
: null,
|
| 581 |
+
);
|
| 582 |
+
// The move being shared goes LAST so its entry (under the
|
| 583 |
+
// stem it's being published as right now) wins over any
|
| 584 |
+
// stale sibling data.
|
| 585 |
+
const localMoves = [
|
| 586 |
+
...(opts.localMoves || []),
|
| 587 |
+
{ ...move, sharedStem: moveStem, fromCommunity: false },
|
| 588 |
+
];
|
| 589 |
+
const rows = buildViewerMetadataRows({ paths: [...predicted], plan, localMoves, existingRows });
|
| 590 |
+
files.push({
|
| 591 |
+
path: plan.metadataPath,
|
| 592 |
+
content: new Blob([buildMetadataJsonl(rows)], { type: "application/json" }),
|
| 593 |
+
});
|
| 594 |
+
}
|
| 595 |
+
} catch (e) {
|
| 596 |
+
// Couldn't determine the repo's state (network flake, 5xx).
|
| 597 |
+
// Ship the move without touching README/metadata: a fresh repo
|
| 598 |
+
// missing its community tag self-heals via the tag-repair sweep
|
| 599 |
+
// next session, and an existing README is left alone rather
|
| 600 |
+
// than risk clobbering a customized one.
|
| 601 |
+
Log.warn("uploadMove: viewer/README housekeeping skipped", { repoId, error: String(e) });
|
| 602 |
+
}
|
| 603 |
+
|
| 604 |
await hfUploadFiles({
|
| 605 |
repo: { type: "dataset", name: repoId },
|
| 606 |
credentials,
|
|
|
|
| 614 |
// files still carry the old slug; drop them so community import
|
| 615 |
// doesn't show the move twice. A failed delete only leaves an orphan
|
| 616 |
// (a harmless duplicate), never data loss.
|
|
|
|
| 617 |
if (oldStem && oldStem !== moveStem) {
|
| 618 |
+
if (!cleanupPlanned) {
|
| 619 |
// Slugs are slugify(label) with no uniqueness: another move
|
| 620 |
// in this folder currently publishes under the old slug, so
|
| 621 |
// `data/<oldStem>.*` is that move's LIVE copy — deleting it
|
|
|
|
| 708 |
m.frameCount = moveJson.set_target_data.length;
|
| 709 |
await putMove(m);
|
| 710 |
}
|
| 711 |
+
// Stale sidecars to drop once the new files are committed: only
|
| 712 |
+
// the .wav files we're replacing with .ogg in this very commit —
|
| 713 |
+
// never a .wav whose move we didn't re-upload (that would orphan
|
| 714 |
+
// its audio). Listed BEFORE the upload so the same snapshot also
|
| 715 |
+
// feeds the viewer config below.
|
| 716 |
+
const upstream = await this.listDatasetFiles(repoId, token);
|
| 717 |
+
const upstreamPaths = upstream.map((f) => f.path);
|
| 718 |
+
const staleWavs = upstreamPaths.filter((p) => {
|
| 719 |
+
const mm = p.match(/^(?:data\/)?(.+)\.wav$/i);
|
| 720 |
+
return mm && audioStems.has(mm[1]);
|
| 721 |
+
});
|
| 722 |
+
|
| 723 |
+
// Viewer support (docs/hf-viewer-fix-spec.md): refresh the README
|
| 724 |
+
// `configs:` key + metadata.jsonl inside the same commit, computed
|
| 725 |
+
// against the post-compress state (new .ogg files in, stale .wav
|
| 726 |
+
// out — rule 4). These repos are owned and often carry customized
|
| 727 |
+
// READMEs, so only ensureViewerConfig ever touches them. Failures
|
| 728 |
+
// are logged and skipped: compression must not be blocked.
|
| 729 |
+
try {
|
| 730 |
+
const stale = new Set(staleWavs);
|
| 731 |
+
const predicted = new Set(upstreamPaths.filter((p) => !stale.has(p)));
|
| 732 |
+
for (const f of files) predicted.add(f.path);
|
| 733 |
+
const plan = viewerConfigPlan([...predicted]);
|
| 734 |
+
if (plan) {
|
| 735 |
+
const existingReadme = await this.fetchRepoText(repoId, "README.md", token);
|
| 736 |
+
const readme = existingReadme === null
|
| 737 |
+
? buildCommunityReadme(repoId.split("/").pop(), repoId, plan.dataFiles)
|
| 738 |
+
: ensureViewerConfig(existingReadme, plan.dataFiles);
|
| 739 |
+
if (readme !== existingReadme) {
|
| 740 |
+
files.push({ path: "README.md", content: new Blob([readme], { type: "text/markdown" }) });
|
| 741 |
+
}
|
| 742 |
+
const existingRows = parseMetadataJsonl(
|
| 743 |
+
predicted.has(plan.metadataPath)
|
| 744 |
+
? await this.fetchRepoText(repoId, plan.metadataPath, token)
|
| 745 |
+
: null,
|
| 746 |
+
);
|
| 747 |
+
// Metadata rows must map by the stems used in THIS commit:
|
| 748 |
+
// compress publishes local moves under slugify(label) (not
|
| 749 |
+
// sharedStem, which may lag behind a local rename).
|
| 750 |
+
const localMoves = moves.map((m) => (m.fromCommunity ? m : { ...m, sharedStem: slugify(m.label) }));
|
| 751 |
+
const rows = buildViewerMetadataRows({
|
| 752 |
+
paths: [...predicted], plan, localMoves, existingRows,
|
| 753 |
+
});
|
| 754 |
+
files.push({
|
| 755 |
+
path: plan.metadataPath,
|
| 756 |
+
content: new Blob([buildMetadataJsonl(rows)], { type: "application/json" }),
|
| 757 |
+
});
|
| 758 |
+
}
|
| 759 |
+
} catch (e) {
|
| 760 |
+
Log.warn("compressAndRepublish: viewer housekeeping skipped", { repoId, error: String(e) });
|
| 761 |
+
}
|
| 762 |
+
|
| 763 |
await hfUploadFiles({
|
| 764 |
repo: { type: "dataset", name: repoId },
|
| 765 |
credentials,
|
| 766 |
files,
|
| 767 |
commitTitle: "Compress: 50 Hz motion + Opus audio",
|
| 768 |
});
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 769 |
if (staleWavs.length) {
|
| 770 |
await hfDeleteFiles({
|
| 771 |
repo: { type: "dataset", name: repoId },
|
|
|
|
| 816 |
},
|
| 817 |
};
|
| 818 |
|
| 819 |
+
// ─── HF dataset-viewer support (docs/hf-viewer-fix-spec.md) ─────────
|
| 820 |
+
//
|
| 821 |
+
// Without a `configs:` key in the README frontmatter, the dataset-viewer
|
| 822 |
+
// infers a loader from file extensions, picks the JSON builder for our
|
| 823 |
+
// motion files, chokes on their shape, and the WHOLE viewer (audio
|
| 824 |
+
// previews included) stays broken. The fix is two additive files —
|
| 825 |
+
// README `configs:` pointing the viewer at audio only, plus a
|
| 826 |
+
// `metadata.jsonl` giving each audio row a title/description — and zero
|
| 827 |
+
// changes to the move files themselves. Verified live on
|
| 828 |
+
// pollen-robotics/reachy-mini-emotions-library (2026-07-03).
|
| 829 |
+
|
| 830 |
+
// Decide what the viewer config should look like for a repo containing
|
| 831 |
+
// `paths` (post-commit). Returns null when the repo has no audio at all
|
| 832 |
+
// (a config whose globs match nothing errors out and re-breaks the
|
| 833 |
+
// viewer — rule 1 of the spec). Repos come in two layouts (files under
|
| 834 |
+
// data/ for JS-app uploads, at the root for Python-seeded libraries);
|
| 835 |
+
// globs are emitted for the layout the audio actually uses, preferring
|
| 836 |
+
// data/ in the (never-seen) mixed case so metadata.jsonl can sit next
|
| 837 |
+
// to the audio it describes.
|
| 838 |
+
export function viewerConfigPlan(paths) {
|
| 839 |
+
const extsByPrefix = { "data/": new Set(), "": new Set() };
|
| 840 |
+
for (const p of paths) {
|
| 841 |
+
const m = /^(data\/)?[^/]+(\.[a-z0-9]+)$/i.exec(p);
|
| 842 |
+
if (!m) continue;
|
| 843 |
+
const ext = m[2].toLowerCase();
|
| 844 |
+
if (!SOUND_EXTENSIONS.includes(ext)) continue;
|
| 845 |
+
extsByPrefix[m[1] || ""].add(ext);
|
| 846 |
+
}
|
| 847 |
+
const prefix = extsByPrefix["data/"].size ? "data/" : "";
|
| 848 |
+
const exts = extsByPrefix[prefix];
|
| 849 |
+
if (!exts.size) return null;
|
| 850 |
+
const metadataPath = `${prefix}metadata.jsonl`;
|
| 851 |
+
const dataFiles = [
|
| 852 |
+
...[...exts].sort().map((e) => `${prefix}*${e}`),
|
| 853 |
+
metadataPath,
|
| 854 |
+
];
|
| 855 |
+
return { prefix, dataFiles, metadataPath };
|
| 856 |
+
}
|
| 857 |
+
|
| 858 |
+
// Render the exact YAML block the viewer needs. Kept as a plain string
|
| 859 |
+
// template (no YAML lib): the shape is fixed and byte-stable output is
|
| 860 |
+
// what makes ensureViewerConfig idempotent.
|
| 861 |
+
export function renderViewerConfigBlock(dataFiles) {
|
| 862 |
+
return [
|
| 863 |
+
"configs:",
|
| 864 |
+
" - config_name: default",
|
| 865 |
+
" data_files:",
|
| 866 |
+
" - split: train",
|
| 867 |
+
" path:",
|
| 868 |
+
...dataFiles.map((p) => ` - "${p}"`),
|
| 869 |
+
].join("\n");
|
| 870 |
+
}
|
| 871 |
+
|
| 872 |
+
// Insert or update ONLY the top-level `configs:` key of a README's YAML
|
| 873 |
+
// frontmatter; every other byte (rest of the frontmatter, the whole
|
| 874 |
+
// body) is preserved verbatim. This is what lets us touch customized,
|
| 875 |
+
// hand-maintained READMEs (e.g. the emotions library) without
|
| 876 |
+
// clobbering them — regenerating the README wholesale is the failure
|
| 877 |
+
// mode this replaces. No audio (empty dataFiles) → README returned
|
| 878 |
+
// untouched.
|
| 879 |
+
export function ensureViewerConfig(readmeText, dataFiles) {
|
| 880 |
+
if (!dataFiles || !dataFiles.length) return readmeText;
|
| 881 |
+
const block = renderViewerConfigBlock(dataFiles);
|
| 882 |
+
const fmMatch = /^---\r?\n([\s\S]*?)\r?\n---(?=\r?\n|$)/.exec(readmeText);
|
| 883 |
+
if (!fmMatch) {
|
| 884 |
+
// No frontmatter at all: prepend one carrying just the config.
|
| 885 |
+
return `---\n${block}\n---\n\n${readmeText}`;
|
| 886 |
+
}
|
| 887 |
+
const fm = fmMatch[1];
|
| 888 |
+
const rest = readmeText.slice(fmMatch[0].length); // starts at the newline after the closing ---
|
| 889 |
+
const lines = fm.split("\n");
|
| 890 |
+
// Locate an existing top-level `configs:` key: from its line to the
|
| 891 |
+
// next top-level key (a line starting with a non-space, non-list,
|
| 892 |
+
// non-comment character) or the end of the frontmatter.
|
| 893 |
+
let start = -1;
|
| 894 |
+
let end = lines.length;
|
| 895 |
+
for (let i = 0; i < lines.length; i++) {
|
| 896 |
+
if (start === -1) {
|
| 897 |
+
if (/^configs:(\s|$)/.test(lines[i])) start = i;
|
| 898 |
+
} else if (/^[^\s#-]/.test(lines[i])) {
|
| 899 |
+
end = i;
|
| 900 |
+
break;
|
| 901 |
+
}
|
| 902 |
+
}
|
| 903 |
+
const newFm = start !== -1
|
| 904 |
+
? [...lines.slice(0, start), block, ...lines.slice(end)].join("\n")
|
| 905 |
+
: `${fm}\n${block}`;
|
| 906 |
+
return `---\n${newFm}\n---${rest}`;
|
| 907 |
+
}
|
| 908 |
+
|
| 909 |
+
// True when a README's frontmatter already carries a top-level
|
| 910 |
+
// `configs:` key — the repair sweep's "already fixed, never clobber"
|
| 911 |
+
// check.
|
| 912 |
+
export function hasViewerConfig(readmeText) {
|
| 913 |
+
const fmMatch = /^---\r?\n([\s\S]*?)\r?\n---(?=\r?\n|$)/.exec(readmeText || "");
|
| 914 |
+
if (!fmMatch) return false;
|
| 915 |
+
return fmMatch[1].split("\n").some((l) => /^configs:(\s|$)/.test(l));
|
| 916 |
+
}
|
| 917 |
+
|
| 918 |
+
// Parse an existing metadata.jsonl into Map(file_name → row) so repair /
|
| 919 |
+
// re-share can keep rows for audio files we know nothing about locally.
|
| 920 |
+
export function parseMetadataJsonl(text) {
|
| 921 |
+
const map = new Map();
|
| 922 |
+
if (!text) return map;
|
| 923 |
+
for (const line of text.split("\n")) {
|
| 924 |
+
const t = line.trim();
|
| 925 |
+
if (!t) continue;
|
| 926 |
+
try {
|
| 927 |
+
const row = JSON.parse(t);
|
| 928 |
+
if (row && typeof row.file_name === "string") map.set(row.file_name, row);
|
| 929 |
+
} catch { /* skip malformed lines, never fail the whole file */ }
|
| 930 |
+
}
|
| 931 |
+
return map;
|
| 932 |
+
}
|
| 933 |
+
|
| 934 |
+
// One metadata row per audio file in the plan's directory. Sources, in
|
| 935 |
+
// priority order: the local IndexedDB move (title = stem, description =
|
| 936 |
+
// what the published move JSON carries), the repo's existing
|
| 937 |
+
// metadata.jsonl row, else title-only. Column names are fixed by the
|
| 938 |
+
// audiofolder loader — and `label` is reserved by it for class labels,
|
| 939 |
+
// so we never emit one.
|
| 940 |
+
export function buildViewerMetadataRows({ paths, plan, localMoves = [], existingRows = new Map() }) {
|
| 941 |
+
const { prefix } = plan;
|
| 942 |
+
const inDir = (p) => (prefix ? p.startsWith(prefix) && !p.slice(prefix.length).includes("/") : !p.includes("/"));
|
| 943 |
+
const base = (p) => (prefix ? p.slice(prefix.length) : p);
|
| 944 |
+
|
| 945 |
+
const jsonStems = new Set();
|
| 946 |
+
const audio = [];
|
| 947 |
+
for (const p of paths) {
|
| 948 |
+
if (!inDir(p)) continue;
|
| 949 |
+
const b = base(p);
|
| 950 |
+
if (b === "marionette.json" || b === "metadata.jsonl") continue;
|
| 951 |
+
if (b.endsWith(".json")) { jsonStems.add(b.slice(0, -".json".length)); continue; }
|
| 952 |
+
const ext = /(\.[a-z0-9]+)$/i.exec(b)?.[1]?.toLowerCase();
|
| 953 |
+
if (ext && SOUND_EXTENSIONS.includes(ext)) audio.push({ fileName: b, stem: b.slice(0, -ext.length) });
|
| 954 |
+
}
|
| 955 |
+
|
| 956 |
+
// Published stem of a local move: imported moves keep their upstream
|
| 957 |
+
// stem (tail of "datasetId/stem" ids), local shares use the slug they
|
| 958 |
+
// were last published under (sharedStem, falling back to the label).
|
| 959 |
+
const byStem = new Map();
|
| 960 |
+
for (const m of localMoves) {
|
| 961 |
+
const stem = m?.fromCommunity && m.id?.includes("/")
|
| 962 |
+
? m.id.split("/").pop()
|
| 963 |
+
: (m?.sharedStem || slugify(m?.label || ""));
|
| 964 |
+
if (stem) byStem.set(stem, m);
|
| 965 |
+
}
|
| 966 |
+
|
| 967 |
+
audio.sort((a, b) => a.fileName.localeCompare(b.fileName));
|
| 968 |
+
return audio.map(({ fileName, stem }) => {
|
| 969 |
+
const local = byStem.get(stem);
|
| 970 |
+
const prev = existingRows.get(fileName);
|
| 971 |
+
const description = local
|
| 972 |
+
? (local.description || local.label || "")
|
| 973 |
+
: (prev?.description || "");
|
| 974 |
+
return {
|
| 975 |
+
file_name: fileName,
|
| 976 |
+
title: stem,
|
| 977 |
+
...(description ? { description } : {}),
|
| 978 |
+
...(jsonStems.has(stem) ? { motion_file: `${stem}.json` } : {}),
|
| 979 |
+
};
|
| 980 |
+
});
|
| 981 |
+
}
|
| 982 |
+
|
| 983 |
+
// Serialize rows to the on-Hub metadata.jsonl format (one JSON object
|
| 984 |
+
// per line). MUST be named metadata.jsonl, never .json — a .json name
|
| 985 |
+
// would be ingested as a move by both the reachy_mini loader (globs
|
| 986 |
+
// *.json) and our own importer (endsWith ".json").
|
| 987 |
+
export function buildMetadataJsonl(rows) {
|
| 988 |
+
return rows.length ? rows.map((r) => JSON.stringify(r)).join("\n") + "\n" : "";
|
| 989 |
+
}
|
| 990 |
+
|
| 991 |
// Build the optional `marionette.json` payload, OR return null when
|
| 992 |
// the folder has no non-default settings worth recording. The schema
|
| 993 |
// version exists so we can add fields later without ambiguity.
|
|
|
|
| 1003 |
// README that carries the community discovery tag in YAML
|
| 1004 |
// frontmatter. Mirrors the format the Python Marionette app uses, so
|
| 1005 |
// datasets published from either app are indistinguishable to the
|
| 1006 |
+
// listing endpoint. `dataFiles` (from viewerConfigPlan) additionally
|
| 1007 |
+
// embeds the dataset-viewer `configs:` block — omitted when the repo
|
| 1008 |
+
// has no audio (an unmatched glob breaks the whole viewer).
|
| 1009 |
+
function buildCommunityReadme(folderName, repoId, dataFiles = null) {
|
| 1010 |
const pretty = (folderName || "marionette").replace(/[-_]+/g, " ");
|
| 1011 |
+
const configs = dataFiles?.length ? `${renderViewerConfigBlock(dataFiles)}\n` : "";
|
| 1012 |
return `---
|
| 1013 |
license: apache-2.0
|
| 1014 |
task_categories:
|
|
|
|
| 1018 |
tags:
|
| 1019 |
- ${HF_COMMUNITY_TAG}
|
| 1020 |
pretty_name: ${pretty} • Reachy Mini Moves
|
| 1021 |
+
${configs}---
|
| 1022 |
|
| 1023 |
# ${pretty} • Reachy Mini Moves
|
| 1024 |
|
|
@@ -38,7 +38,7 @@ import { connectToHost } from "https://cdn.jsdelivr.net/npm/@pollen-robotics/rea
|
|
| 38 |
import { state, setRenderImpl } from "./lib/state.js";
|
| 39 |
import { Log } from "./lib/log.js";
|
| 40 |
import { busy, runCleanups } from "./lib/busy.js";
|
| 41 |
-
import { ensureDefaultDataset, listSharedMoves, countAllMoves } from "./lib/db.js";
|
| 42 |
import { HF } from "./lib/hf.js";
|
| 43 |
import { runMigrationWithOverlay, runMigrationIfNeeded } from "./lib/audio-migrate.js";
|
| 44 |
import { createMotorSubsystem, flat16 } from "./lib/motor.js";
|
|
@@ -366,6 +366,14 @@ async function bootEmbed() {
|
|
| 366 |
Log.warn("community tag migration failed", { error: String(e) }),
|
| 367 |
);
|
| 368 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 369 |
// Audio re-upload retry: any shared moves that needed re-upload
|
| 370 |
// during bootstrap get flushed now that we have a token. Background;
|
| 371 |
// no overlay.
|
|
@@ -504,6 +512,56 @@ async function runStartupGreeting(robot, Audio_, Playback) {
|
|
| 504 |
// current tags; if we own it and the tag is missing, push a README
|
| 505 |
// to fix it. We don't touch already-tagged repos (don't clobber
|
| 506 |
// user-customised READMEs).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 507 |
let _tagMigrationRan = false;
|
| 508 |
async function runCommunityTagMigration(robot) {
|
| 509 |
if (_tagMigrationRan) return;
|
|
|
|
| 38 |
import { state, setRenderImpl } from "./lib/state.js";
|
| 39 |
import { Log } from "./lib/log.js";
|
| 40 |
import { busy, runCleanups } from "./lib/busy.js";
|
| 41 |
+
import { ensureDefaultDataset, listSharedMoves, countAllMoves, kvGet, kvSet } from "./lib/db.js";
|
| 42 |
import { HF } from "./lib/hf.js";
|
| 43 |
import { runMigrationWithOverlay, runMigrationIfNeeded } from "./lib/audio-migrate.js";
|
| 44 |
import { createMotorSubsystem, flat16 } from "./lib/motor.js";
|
|
|
|
| 366 |
Log.warn("community tag migration failed", { error: String(e) }),
|
| 367 |
);
|
| 368 |
|
| 369 |
+
// One-shot dataset-viewer repair: make every owned dataset playable
|
| 370 |
+
// on its HF page (README `configs:` + metadata.jsonl). Persisted in
|
| 371 |
+
// IndexedDB so it doesn't re-run every session; new shares emit the
|
| 372 |
+
// fix themselves.
|
| 373 |
+
runViewerConfigMigration(robot).catch((e) =>
|
| 374 |
+
Log.warn("viewer config migration failed", { error: String(e) }),
|
| 375 |
+
);
|
| 376 |
+
|
| 377 |
// Audio re-upload retry: any shared moves that needed re-upload
|
| 378 |
// during bootstrap get flushed now that we have a token. Background;
|
| 379 |
// no overlay.
|
|
|
|
| 512 |
// current tags; if we own it and the tag is missing, push a README
|
| 513 |
// to fix it. We don't touch already-tagged repos (don't clobber
|
| 514 |
// user-customised READMEs).
|
| 515 |
+
// Dataset-viewer repair sweep (docs/hf-viewer-fix-spec.md): every repo
|
| 516 |
+
// the user owns gets the README `configs:` + metadata.jsonl treatment so
|
| 517 |
+
// its HF page shows playable audio rows. Runs automatically once we have
|
| 518 |
+
// a token, then records completion in IndexedDB (kv) so later sessions
|
| 519 |
+
// skip it entirely — the per-repo check is idempotent anyway (repos
|
| 520 |
+
// whose README already carries `configs:` are never touched), and every
|
| 521 |
+
// new/updated share emits the fix by itself. The flag is only written
|
| 522 |
+
// when the sweep finishes with zero errors, so a flaky run retries next
|
| 523 |
+
// session. Bump the kv key's version suffix to force a global re-sweep
|
| 524 |
+
// after a future format change.
|
| 525 |
+
const VIEWER_MIGRATION_KV = "viewer_config_migration_v1";
|
| 526 |
+
let _viewerMigrationRan = false;
|
| 527 |
+
async function runViewerConfigMigration(robot) {
|
| 528 |
+
if (_viewerMigrationRan) return;
|
| 529 |
+
_viewerMigrationRan = true;
|
| 530 |
+
if (!robot._token) return;
|
| 531 |
+
if (await kvGet(VIEWER_MIGRATION_KV)) return;
|
| 532 |
+
|
| 533 |
+
// Group the locally-shared moves by the repo they live in; they're
|
| 534 |
+
// the metadata title/description source. Owned-namespace filtering
|
| 535 |
+
// happens inside the sweep (needs the whoAmI namespace).
|
| 536 |
+
const moves = await listSharedMoves();
|
| 537 |
+
const repos = new Map();
|
| 538 |
+
for (const m of moves) {
|
| 539 |
+
const repoId = m.sharedUrl?.split("/datasets/")[1]?.split(/[?#]/)[0];
|
| 540 |
+
if (!repoId) continue;
|
| 541 |
+
if (!repos.has(repoId)) repos.set(repoId, []);
|
| 542 |
+
repos.get(repoId).push(m);
|
| 543 |
+
}
|
| 544 |
+
if (!repos.size) {
|
| 545 |
+
// Nothing shared from this browser — nothing to repair, and
|
| 546 |
+
// nothing to gain from re-checking every session.
|
| 547 |
+
await kvSet(VIEWER_MIGRATION_KV, { done: true, at: Date.now(), checked: 0, fixed: 0 });
|
| 548 |
+
return;
|
| 549 |
+
}
|
| 550 |
+
|
| 551 |
+
const { checked, fixed, errors } = await HF.repairOwnedViewerConfigs({
|
| 552 |
+
token: robot._token,
|
| 553 |
+
repos,
|
| 554 |
+
});
|
| 555 |
+
Log.info("viewer config migration", { checked, fixed, errors });
|
| 556 |
+
if (fixed > 0) {
|
| 557 |
+
const s = fixed === 1 ? "" : "s";
|
| 558 |
+
toast(`Enabled HF audio previews on ${fixed} dataset${s}`, "success");
|
| 559 |
+
}
|
| 560 |
+
if (errors === 0) {
|
| 561 |
+
await kvSet(VIEWER_MIGRATION_KV, { done: true, at: Date.now(), checked, fixed });
|
| 562 |
+
}
|
| 563 |
+
}
|
| 564 |
+
|
| 565 |
let _tagMigrationRan = false;
|
| 566 |
async function runCommunityTagMigration(robot) {
|
| 567 |
if (_tagMigrationRan) return;
|
|
@@ -36,6 +36,7 @@ function makeMove(overrides = {}) {
|
|
| 36 |
|
| 37 |
const origNamespace = HF.getHfNamespace;
|
| 38 |
const origList = HF.listDatasetFiles;
|
|
|
|
| 39 |
|
| 40 |
beforeEach(() => {
|
| 41 |
resetHubStub();
|
|
@@ -46,11 +47,15 @@ beforeEach(() => {
|
|
| 46 |
{ path: "data/dance.ogg", _prefix: "data" },
|
| 47 |
{ path: "data/other.json", _prefix: "data" },
|
| 48 |
];
|
|
|
|
|
|
|
|
|
|
| 49 |
});
|
| 50 |
|
| 51 |
afterEach(() => {
|
| 52 |
HF.getHfNamespace = origNamespace;
|
| 53 |
HF.listDatasetFiles = origList;
|
|
|
|
| 54 |
});
|
| 55 |
|
| 56 |
describe("uploadMove rename cleanup", () => {
|
|
@@ -128,3 +133,110 @@ describe("uploadMove rename cleanup", () => {
|
|
| 128 |
expect(paths).toContain("data/dance-2.ogg");
|
| 129 |
});
|
| 130 |
});
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 36 |
|
| 37 |
const origNamespace = HF.getHfNamespace;
|
| 38 |
const origList = HF.listDatasetFiles;
|
| 39 |
+
const origFetchText = HF.fetchRepoText;
|
| 40 |
|
| 41 |
beforeEach(() => {
|
| 42 |
resetHubStub();
|
|
|
|
| 47 |
{ path: "data/dance.ogg", _prefix: "data" },
|
| 48 |
{ path: "data/other.json", _prefix: "data" },
|
| 49 |
];
|
| 50 |
+
// No README / metadata upstream by default (fresh-repo behavior);
|
| 51 |
+
// also keeps unit tests off the real network.
|
| 52 |
+
HF.fetchRepoText = async () => null;
|
| 53 |
});
|
| 54 |
|
| 55 |
afterEach(() => {
|
| 56 |
HF.getHfNamespace = origNamespace;
|
| 57 |
HF.listDatasetFiles = origList;
|
| 58 |
+
HF.fetchRepoText = origFetchText;
|
| 59 |
});
|
| 60 |
|
| 61 |
describe("uploadMove rename cleanup", () => {
|
|
|
|
| 133 |
expect(paths).toContain("data/dance-2.ogg");
|
| 134 |
});
|
| 135 |
});
|
| 136 |
+
|
| 137 |
+
describe("uploadMove viewer support", () => {
|
| 138 |
+
const fileText = async (call, path) => {
|
| 139 |
+
const f = call.arg.files.find((x) => x.path === path);
|
| 140 |
+
return f ? f.content.text() : null;
|
| 141 |
+
};
|
| 142 |
+
|
| 143 |
+
it("fresh repo: README carries the community tag AND the configs block; metadata covers all audio", async () => {
|
| 144 |
+
const move = makeMove();
|
| 145 |
+
await HF.uploadMove(move, "My Folder", "tok", { localMoves: [] });
|
| 146 |
+
|
| 147 |
+
const up = calls.find((c) => c.fn === "uploadFiles");
|
| 148 |
+
const readme = await fileText(up, "README.md");
|
| 149 |
+
expect(readme).toContain("reachy_mini_community_moves");
|
| 150 |
+
expect(readme).toContain("configs:");
|
| 151 |
+
expect(readme).toContain(`- "data/*.ogg"`);
|
| 152 |
+
expect(readme).toContain(`- "data/metadata.jsonl"`);
|
| 153 |
+
|
| 154 |
+
const meta = await fileText(up, "data/metadata.jsonl");
|
| 155 |
+
// Rename cleanup will delete data/dance.* right after this commit
|
| 156 |
+
// — the metadata must NOT reference them (rule 4), only the new
|
| 157 |
+
// stem's audio.
|
| 158 |
+
expect(meta).toContain(`"file_name":"dance-2.ogg"`);
|
| 159 |
+
expect(meta).not.toContain(`"file_name":"dance.ogg"`);
|
| 160 |
+
expect(meta).toContain(`"motion_file":"dance-2.json"`);
|
| 161 |
+
});
|
| 162 |
+
|
| 163 |
+
it("existing README: only the configs key is inserted, body preserved", async () => {
|
| 164 |
+
const custom = `---
|
| 165 |
+
license: apache-2.0
|
| 166 |
+
tags:
|
| 167 |
+
- reachy_mini_community_moves
|
| 168 |
+
---
|
| 169 |
+
|
| 170 |
+
# Hand-written title
|
| 171 |
+
|
| 172 |
+
Precious hand-written body.
|
| 173 |
+
`;
|
| 174 |
+
HF.fetchRepoText = async (repoId, path) => (path === "README.md" ? custom : null);
|
| 175 |
+
const move = makeMove({ sharedStem: "dance-2" }); // no rename
|
| 176 |
+
await HF.uploadMove(move, "My Folder", "tok");
|
| 177 |
+
|
| 178 |
+
const up = calls.find((c) => c.fn === "uploadFiles");
|
| 179 |
+
const readme = await fileText(up, "README.md");
|
| 180 |
+
expect(readme).toContain("Precious hand-written body.");
|
| 181 |
+
expect(readme).toContain("# Hand-written title");
|
| 182 |
+
expect(readme).toContain("configs:");
|
| 183 |
+
});
|
| 184 |
+
|
| 185 |
+
it("README already viewer-ready and unchanged → not re-uploaded", async () => {
|
| 186 |
+
const move = makeMove({ sharedStem: "dance-2" });
|
| 187 |
+
// First share produces the ensured README; feed it back as the
|
| 188 |
+
// upstream one — the second share must skip the README write.
|
| 189 |
+
await HF.uploadMove(makeMove({ sharedStem: "dance-2" }), "My Folder", "tok");
|
| 190 |
+
const first = calls.find((c) => c.fn === "uploadFiles");
|
| 191 |
+
const ensured = await fileText(first, "README.md");
|
| 192 |
+
|
| 193 |
+
resetHubStub();
|
| 194 |
+
HF.fetchRepoText = async (repoId, path) => (path === "README.md" ? ensured : null);
|
| 195 |
+
await HF.uploadMove(move, "My Folder", "tok");
|
| 196 |
+
const up = calls.find((c) => c.fn === "uploadFiles");
|
| 197 |
+
expect(up.arg.files.map((f) => f.path)).not.toContain("README.md");
|
| 198 |
+
});
|
| 199 |
+
|
| 200 |
+
it("silent-only repo (no audio anywhere): no configs, no metadata.jsonl", async () => {
|
| 201 |
+
HF.listDatasetFiles = async () => [{ path: "data/other.json", _prefix: "data" }];
|
| 202 |
+
const move = makeMove({ audioBlob: null, audioMime: null, sharedStem: "dance-2" });
|
| 203 |
+
await HF.uploadMove(move, "My Folder", "tok");
|
| 204 |
+
|
| 205 |
+
const up = calls.find((c) => c.fn === "uploadFiles");
|
| 206 |
+
const paths = up.arg.files.map((f) => f.path);
|
| 207 |
+
expect(paths).not.toContain("data/metadata.jsonl");
|
| 208 |
+
const readme = await fileText(up, "README.md");
|
| 209 |
+
expect(readme).toContain("reachy_mini_community_moves");
|
| 210 |
+
expect(readme).not.toContain("configs:");
|
| 211 |
+
});
|
| 212 |
+
|
| 213 |
+
it("repo-state fetch failure: the share still goes through, README untouched", async () => {
|
| 214 |
+
HF.fetchRepoText = async () => { throw new Error("HF 500"); };
|
| 215 |
+
const move = makeMove({ sharedStem: "dance-2" });
|
| 216 |
+
const url = await HF.uploadMove(move, "My Folder", "tok");
|
| 217 |
+
|
| 218 |
+
expect(url).toContain("testuser/my-folder");
|
| 219 |
+
const up = calls.find((c) => c.fn === "uploadFiles");
|
| 220 |
+
const paths = up.arg.files.map((f) => f.path);
|
| 221 |
+
expect(paths).toContain("data/dance-2.json");
|
| 222 |
+
expect(paths).not.toContain("README.md");
|
| 223 |
+
expect(paths).not.toContain("data/metadata.jsonl");
|
| 224 |
+
});
|
| 225 |
+
|
| 226 |
+
it("sibling moves' descriptions land in metadata via opts.localMoves", async () => {
|
| 227 |
+
HF.listDatasetFiles = async () => [
|
| 228 |
+
{ path: "data/other.json", _prefix: "data" },
|
| 229 |
+
{ path: "data/other.ogg", _prefix: "data" },
|
| 230 |
+
];
|
| 231 |
+
const move = makeMove({ sharedStem: "dance-2" });
|
| 232 |
+
await HF.uploadMove(move, "My Folder", "tok", {
|
| 233 |
+
localMoves: [{ label: "Other", sharedStem: "other", description: "The other move." }],
|
| 234 |
+
});
|
| 235 |
+
|
| 236 |
+
const up = calls.find((c) => c.fn === "uploadFiles");
|
| 237 |
+
const meta = await fileText(up, "data/metadata.jsonl");
|
| 238 |
+
expect(meta).toContain(`"file_name":"dance-2.ogg"`);
|
| 239 |
+
expect(meta).toContain(`"file_name":"other.ogg"`);
|
| 240 |
+
expect(meta).toContain(`"description":"The other move."`);
|
| 241 |
+
});
|
| 242 |
+
});
|
|
@@ -0,0 +1,116 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// repairOwnedViewerConfigs — the startup auto-repair sweep. It pushes
|
| 2 |
+
// commits unattended, so its guards get pinned here: own-namespace only,
|
| 3 |
+
// skip already-fixed repos, skip no-audio repos, byte-preserving README
|
| 4 |
+
// edit, per-repo error isolation.
|
| 5 |
+
|
| 6 |
+
import { describe, it, expect, beforeEach, afterEach } from "vitest";
|
| 7 |
+
import { HF } from "../../lib/hf.js";
|
| 8 |
+
import { calls, behavior, resetHubStub } from "../stubs/hf-hub.js";
|
| 9 |
+
|
| 10 |
+
const README = `---
|
| 11 |
+
license: apache-2.0
|
| 12 |
+
tags:
|
| 13 |
+
- reachy_mini_community_moves
|
| 14 |
+
---
|
| 15 |
+
|
| 16 |
+
# Vintage dataset
|
| 17 |
+
|
| 18 |
+
Hand-written notes.
|
| 19 |
+
`;
|
| 20 |
+
|
| 21 |
+
const origNamespace = HF.getHfNamespace;
|
| 22 |
+
const origList = HF.listDatasetFiles;
|
| 23 |
+
const origFetchText = HF.fetchRepoText;
|
| 24 |
+
|
| 25 |
+
beforeEach(() => {
|
| 26 |
+
resetHubStub();
|
| 27 |
+
HF.getHfNamespace = async () => "testuser";
|
| 28 |
+
HF.listDatasetFiles = async () => [
|
| 29 |
+
{ path: "data/wave.json", _prefix: "data" },
|
| 30 |
+
{ path: "data/wave.ogg", _prefix: "data" },
|
| 31 |
+
];
|
| 32 |
+
HF.fetchRepoText = async (repoId, path) => (path === "README.md" ? README : null);
|
| 33 |
+
});
|
| 34 |
+
|
| 35 |
+
afterEach(() => {
|
| 36 |
+
HF.getHfNamespace = origNamespace;
|
| 37 |
+
HF.listDatasetFiles = origList;
|
| 38 |
+
HF.fetchRepoText = origFetchText;
|
| 39 |
+
});
|
| 40 |
+
|
| 41 |
+
const repoMap = (entries) => new Map(entries);
|
| 42 |
+
|
| 43 |
+
describe("repairOwnedViewerConfigs", () => {
|
| 44 |
+
it("fixes an owned repo missing the config: one commit, README preserved + metadata", async () => {
|
| 45 |
+
const res = await HF.repairOwnedViewerConfigs({
|
| 46 |
+
token: "tok",
|
| 47 |
+
repos: repoMap([["testuser/my-moves", [{ label: "Wave", sharedStem: "wave", description: "Hi!" }]]]),
|
| 48 |
+
});
|
| 49 |
+
expect(res).toEqual({ checked: 1, fixed: 1, errors: 0 });
|
| 50 |
+
|
| 51 |
+
const up = calls.find((c) => c.fn === "uploadFiles");
|
| 52 |
+
expect(up.arg.commitTitle).toBe("Marionette: enable dataset-viewer audio previews");
|
| 53 |
+
const readme = await up.arg.files.find((f) => f.path === "README.md").content.text();
|
| 54 |
+
expect(readme).toContain("Hand-written notes.");
|
| 55 |
+
expect(readme).toContain("configs:");
|
| 56 |
+
expect(readme).toContain(`- "data/*.ogg"`);
|
| 57 |
+
const meta = await up.arg.files.find((f) => f.path === "data/metadata.jsonl").content.text();
|
| 58 |
+
expect(meta).toContain(`"file_name":"wave.ogg"`);
|
| 59 |
+
expect(meta).toContain(`"description":"Hi!"`);
|
| 60 |
+
});
|
| 61 |
+
|
| 62 |
+
it("skips repos outside the user's namespace entirely", async () => {
|
| 63 |
+
const res = await HF.repairOwnedViewerConfigs({
|
| 64 |
+
token: "tok",
|
| 65 |
+
repos: repoMap([["pollen-robotics/emotions", []]]),
|
| 66 |
+
});
|
| 67 |
+
expect(res).toEqual({ checked: 0, fixed: 0, errors: 0 });
|
| 68 |
+
expect(calls.filter((c) => c.fn === "uploadFiles")).toHaveLength(0);
|
| 69 |
+
});
|
| 70 |
+
|
| 71 |
+
it("skips repos whose README already has configs (idempotent)", async () => {
|
| 72 |
+
HF.fetchRepoText = async (repoId, path) =>
|
| 73 |
+
(path === "README.md" ? `---\nconfigs:\n - x\n---\nbody` : null);
|
| 74 |
+
const res = await HF.repairOwnedViewerConfigs({
|
| 75 |
+
token: "tok",
|
| 76 |
+
repos: repoMap([["testuser/my-moves", []]]),
|
| 77 |
+
});
|
| 78 |
+
expect(res).toEqual({ checked: 1, fixed: 0, errors: 0 });
|
| 79 |
+
expect(calls.filter((c) => c.fn === "uploadFiles")).toHaveLength(0);
|
| 80 |
+
});
|
| 81 |
+
|
| 82 |
+
it("skips repos with no audio (a config there would break the viewer)", async () => {
|
| 83 |
+
HF.listDatasetFiles = async () => [{ path: "data/silent.json", _prefix: "data" }];
|
| 84 |
+
const res = await HF.repairOwnedViewerConfigs({
|
| 85 |
+
token: "tok",
|
| 86 |
+
repos: repoMap([["testuser/my-moves", []]]),
|
| 87 |
+
});
|
| 88 |
+
expect(res).toEqual({ checked: 1, fixed: 0, errors: 0 });
|
| 89 |
+
expect(calls.filter((c) => c.fn === "uploadFiles")).toHaveLength(0);
|
| 90 |
+
});
|
| 91 |
+
|
| 92 |
+
it("isolates per-repo failures and keeps sweeping", async () => {
|
| 93 |
+
let call = 0;
|
| 94 |
+
behavior.uploadFiles = async () => { if (++call === 1) throw new Error("boom"); };
|
| 95 |
+
const res = await HF.repairOwnedViewerConfigs({
|
| 96 |
+
token: "tok",
|
| 97 |
+
repos: repoMap([
|
| 98 |
+
["testuser/broken", []],
|
| 99 |
+
["testuser/fine", []],
|
| 100 |
+
]),
|
| 101 |
+
});
|
| 102 |
+
expect(res).toEqual({ checked: 2, fixed: 1, errors: 1 });
|
| 103 |
+
});
|
| 104 |
+
|
| 105 |
+
it("generates a community README when the repo has none", async () => {
|
| 106 |
+
HF.fetchRepoText = async () => null;
|
| 107 |
+
await HF.repairOwnedViewerConfigs({
|
| 108 |
+
token: "tok",
|
| 109 |
+
repos: repoMap([["testuser/my-moves", []]]),
|
| 110 |
+
});
|
| 111 |
+
const up = calls.find((c) => c.fn === "uploadFiles");
|
| 112 |
+
const readme = await up.arg.files.find((f) => f.path === "README.md").content.text();
|
| 113 |
+
expect(readme).toContain("reachy_mini_community_moves");
|
| 114 |
+
expect(readme).toContain("configs:");
|
| 115 |
+
});
|
| 116 |
+
});
|
|
@@ -0,0 +1,184 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Dataset-viewer support helpers (docs/hf-viewer-fix-spec.md):
|
| 2 |
+
// viewerConfigPlan (glob planning per layout/extension), ensureViewerConfig
|
| 3 |
+
// (byte-preserving frontmatter surgery), metadata.jsonl building/parsing.
|
| 4 |
+
|
| 5 |
+
import { describe, it, expect } from "vitest";
|
| 6 |
+
import {
|
| 7 |
+
viewerConfigPlan,
|
| 8 |
+
renderViewerConfigBlock,
|
| 9 |
+
ensureViewerConfig,
|
| 10 |
+
hasViewerConfig,
|
| 11 |
+
parseMetadataJsonl,
|
| 12 |
+
buildViewerMetadataRows,
|
| 13 |
+
buildMetadataJsonl,
|
| 14 |
+
} from "../../lib/hf.js";
|
| 15 |
+
|
| 16 |
+
describe("viewerConfigPlan", () => {
|
| 17 |
+
it("emits one glob per audio extension present, data/ layout", () => {
|
| 18 |
+
const plan = viewerConfigPlan([
|
| 19 |
+
"data/a.json", "data/a.ogg", "data/b.json", "data/b.wav", "README.md",
|
| 20 |
+
]);
|
| 21 |
+
expect(plan.prefix).toBe("data/");
|
| 22 |
+
expect(plan.dataFiles).toEqual(["data/*.ogg", "data/*.wav", "data/metadata.jsonl"]);
|
| 23 |
+
expect(plan.metadataPath).toBe("data/metadata.jsonl");
|
| 24 |
+
});
|
| 25 |
+
|
| 26 |
+
it("supports root layout (Python-seeded libraries)", () => {
|
| 27 |
+
const plan = viewerConfigPlan(["a.json", "a.ogg", "README.md"]);
|
| 28 |
+
expect(plan.prefix).toBe("");
|
| 29 |
+
expect(plan.dataFiles).toEqual(["*.ogg", "metadata.jsonl"]);
|
| 30 |
+
});
|
| 31 |
+
|
| 32 |
+
it("returns null when the repo has no audio (rule 1: unmatched globs break the viewer)", () => {
|
| 33 |
+
expect(viewerConfigPlan(["data/a.json", "README.md", "marionette.json"])).toBeNull();
|
| 34 |
+
expect(viewerConfigPlan([])).toBeNull();
|
| 35 |
+
});
|
| 36 |
+
|
| 37 |
+
it("never treats .json/.jsonl or non-audio files as audio", () => {
|
| 38 |
+
const plan = viewerConfigPlan(["data/a.ogg", "data/metadata.jsonl", "data/b.json", "data/c.txt"]);
|
| 39 |
+
expect(plan.dataFiles).toEqual(["data/*.ogg", "data/metadata.jsonl"]);
|
| 40 |
+
});
|
| 41 |
+
|
| 42 |
+
it("prefers the data/ layout when both layouts hold audio", () => {
|
| 43 |
+
const plan = viewerConfigPlan(["rogue.wav", "data/a.ogg"]);
|
| 44 |
+
expect(plan.prefix).toBe("data/");
|
| 45 |
+
expect(plan.dataFiles).toEqual(["data/*.ogg", "data/metadata.jsonl"]);
|
| 46 |
+
});
|
| 47 |
+
});
|
| 48 |
+
|
| 49 |
+
describe("ensureViewerConfig", () => {
|
| 50 |
+
const FILES = ["data/*.ogg", "data/metadata.jsonl"];
|
| 51 |
+
const README = `---
|
| 52 |
+
license: apache-2.0
|
| 53 |
+
tags:
|
| 54 |
+
- reachy_mini_community_moves
|
| 55 |
+
pretty_name: My Moves • Reachy Mini Moves
|
| 56 |
+
---
|
| 57 |
+
|
| 58 |
+
# My Moves
|
| 59 |
+
|
| 60 |
+
Hand-written body that must never change.
|
| 61 |
+
`;
|
| 62 |
+
|
| 63 |
+
it("inserts a configs block into existing frontmatter, preserving everything else", () => {
|
| 64 |
+
const out = ensureViewerConfig(README, FILES);
|
| 65 |
+
expect(hasViewerConfig(out)).toBe(true);
|
| 66 |
+
expect(out).toContain(`- "data/*.ogg"`);
|
| 67 |
+
expect(out).toContain(`- "data/metadata.jsonl"`);
|
| 68 |
+
// Byte-for-byte preservation outside the block: removing the block
|
| 69 |
+
// lines must give back the original.
|
| 70 |
+
const stripped = out.split("\n")
|
| 71 |
+
.filter((l, i, arr) => {
|
| 72 |
+
const block = renderViewerConfigBlock(FILES).split("\n");
|
| 73 |
+
const start = arr.indexOf("configs:");
|
| 74 |
+
return i < start || i >= start + block.length;
|
| 75 |
+
})
|
| 76 |
+
.join("\n");
|
| 77 |
+
expect(stripped).toBe(README);
|
| 78 |
+
});
|
| 79 |
+
|
| 80 |
+
it("is idempotent", () => {
|
| 81 |
+
const once = ensureViewerConfig(README, FILES);
|
| 82 |
+
expect(ensureViewerConfig(once, FILES)).toBe(once);
|
| 83 |
+
});
|
| 84 |
+
|
| 85 |
+
it("updates an existing configs block in place (new extensions appear)", () => {
|
| 86 |
+
const once = ensureViewerConfig(README, ["data/*.wav", "data/metadata.jsonl"]);
|
| 87 |
+
const twice = ensureViewerConfig(once, FILES);
|
| 88 |
+
expect(twice).not.toContain(`"data/*.wav"`);
|
| 89 |
+
expect(twice).toContain(`"data/*.ogg"`);
|
| 90 |
+
// Still exactly one configs key.
|
| 91 |
+
expect(twice.match(/^configs:/gm)).toHaveLength(1);
|
| 92 |
+
});
|
| 93 |
+
|
| 94 |
+
it("replaces a configs block that sits before other keys, keeping them", () => {
|
| 95 |
+
const readme = `---
|
| 96 |
+
configs:
|
| 97 |
+
- config_name: default
|
| 98 |
+
data_files:
|
| 99 |
+
- split: train
|
| 100 |
+
path:
|
| 101 |
+
- "data/*.wav"
|
| 102 |
+
tags:
|
| 103 |
+
- reachy_mini_community_moves
|
| 104 |
+
---
|
| 105 |
+
|
| 106 |
+
body
|
| 107 |
+
`;
|
| 108 |
+
const out = ensureViewerConfig(readme, FILES);
|
| 109 |
+
expect(out).toContain("tags:\n - reachy_mini_community_moves");
|
| 110 |
+
expect(out).toContain(`"data/*.ogg"`);
|
| 111 |
+
expect(out).not.toContain(`"data/*.wav"`);
|
| 112 |
+
expect(out).toContain("\nbody\n");
|
| 113 |
+
});
|
| 114 |
+
|
| 115 |
+
it("returns the README untouched when there is no audio", () => {
|
| 116 |
+
expect(ensureViewerConfig(README, [])).toBe(README);
|
| 117 |
+
expect(ensureViewerConfig(README, null)).toBe(README);
|
| 118 |
+
});
|
| 119 |
+
|
| 120 |
+
it("prepends frontmatter when the README has none", () => {
|
| 121 |
+
const out = ensureViewerConfig("# Just a body\n", FILES);
|
| 122 |
+
expect(out.startsWith("---\nconfigs:")).toBe(true);
|
| 123 |
+
expect(out.endsWith("# Just a body\n")).toBe(true);
|
| 124 |
+
});
|
| 125 |
+
});
|
| 126 |
+
|
| 127 |
+
describe("hasViewerConfig", () => {
|
| 128 |
+
it("detects a top-level configs key only", () => {
|
| 129 |
+
expect(hasViewerConfig("---\nconfigs:\n - x\n---\nbody")).toBe(true);
|
| 130 |
+
expect(hasViewerConfig("---\ntags:\n - configs: nope\n---\n")).toBe(false);
|
| 131 |
+
expect(hasViewerConfig("no frontmatter, mentions configs:")).toBe(false);
|
| 132 |
+
expect(hasViewerConfig("")).toBe(false);
|
| 133 |
+
});
|
| 134 |
+
});
|
| 135 |
+
|
| 136 |
+
describe("metadata.jsonl", () => {
|
| 137 |
+
it("parses rows and skips malformed lines", () => {
|
| 138 |
+
const map = parseMetadataJsonl(
|
| 139 |
+
`{"file_name": "a.ogg", "title": "a"}\nnot json\n\n{"file_name": "b.ogg", "description": "d"}\n`,
|
| 140 |
+
);
|
| 141 |
+
expect([...map.keys()]).toEqual(["a.ogg", "b.ogg"]);
|
| 142 |
+
expect(map.get("b.ogg").description).toBe("d");
|
| 143 |
+
});
|
| 144 |
+
|
| 145 |
+
it("builds rows from local moves, existing rows, and stems — no `label` column ever", () => {
|
| 146 |
+
const paths = ["data/dance.json", "data/dance.ogg", "data/mystery.wav", "data/silent.json"];
|
| 147 |
+
const plan = viewerConfigPlan(paths);
|
| 148 |
+
const rows = buildViewerMetadataRows({
|
| 149 |
+
paths,
|
| 150 |
+
plan,
|
| 151 |
+
localMoves: [
|
| 152 |
+
{ label: "Dance", sharedStem: "dance", description: "A great dance." },
|
| 153 |
+
],
|
| 154 |
+
existingRows: new Map([["mystery.wav", { file_name: "mystery.wav", description: "kept from before" }]]),
|
| 155 |
+
});
|
| 156 |
+
expect(rows).toEqual([
|
| 157 |
+
{ file_name: "dance.ogg", title: "dance", description: "A great dance.", motion_file: "dance.json" },
|
| 158 |
+
{ file_name: "mystery.wav", title: "mystery", description: "kept from before" },
|
| 159 |
+
]);
|
| 160 |
+
for (const r of rows) expect("label" in r).toBe(false);
|
| 161 |
+
});
|
| 162 |
+
|
| 163 |
+
it("maps imported moves by their upstream stem (id tail)", () => {
|
| 164 |
+
const paths = ["data/amazed1.json", "data/amazed1.ogg"];
|
| 165 |
+
const plan = viewerConfigPlan(paths);
|
| 166 |
+
const rows = buildViewerMetadataRows({
|
| 167 |
+
paths,
|
| 168 |
+
plan,
|
| 169 |
+
localMoves: [{ id: "someone/repo/amazed1", fromCommunity: true, label: "Amazed", description: "Wow." }],
|
| 170 |
+
});
|
| 171 |
+
expect(rows).toEqual([
|
| 172 |
+
{ file_name: "amazed1.ogg", title: "amazed1", description: "Wow.", motion_file: "amazed1.json" },
|
| 173 |
+
]);
|
| 174 |
+
});
|
| 175 |
+
|
| 176 |
+
it("serializes one JSON object per line with a trailing newline", () => {
|
| 177 |
+
const text = buildMetadataJsonl([
|
| 178 |
+
{ file_name: "a.ogg", title: "a" },
|
| 179 |
+
{ file_name: "b.ogg", title: "b" },
|
| 180 |
+
]);
|
| 181 |
+
expect(text).toBe(`{"file_name":"a.ogg","title":"a"}\n{"file_name":"b.ogg","title":"b"}\n`);
|
| 182 |
+
expect(buildMetadataJsonl([])).toBe("");
|
| 183 |
+
});
|
| 184 |
+
});
|
|
@@ -686,6 +686,10 @@ export function createFolder({
|
|
| 686 |
const url = await HF.uploadMove(m, repoName, robot._token, {
|
| 687 |
audioOffsetMs: folder?.audioOffsetMs,
|
| 688 |
protectedStems,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 689 |
});
|
| 690 |
m.shared = true;
|
| 691 |
m.sharedUrl = url;
|
|
|
|
| 686 |
const url = await HF.uploadMove(m, repoName, robot._token, {
|
| 687 |
audioOffsetMs: folder?.audioOffsetMs,
|
| 688 |
protectedStems,
|
| 689 |
+
// Feed the folder's moves to the metadata.jsonl
|
| 690 |
+
// builder so every audio row in the repo keeps a
|
| 691 |
+
// title/description, not just the move being shared.
|
| 692 |
+
localMoves: siblings,
|
| 693 |
});
|
| 694 |
m.shared = true;
|
| 695 |
m.sharedUrl = url;
|