RemiFabre commited on
Commit
2a4933f
·
1 Parent(s): 2b2b9a5

HF dataset-viewer fix: configs + metadata.jsonl on share/compress, auto-repair sweep at startup

Browse files

Implements docs/hf-viewer-fix-spec.md (root cause: extension-inferred
JSON loader chokes on motion files and disables the whole viewer).

Share & compress paths (deliverable A):
- viewerConfigPlan/renderViewerConfigBlock/ensureViewerConfig/
buildViewerMetadataRows/buildMetadataJsonl/parseMetadataJsonl helpers
(exported for tests); ensureViewerConfig only touches the frontmatter's
configs: key — byte-preserving, idempotent
- uploadMove: README is no longer regenerated on every share (the old
clobber-customized-READMEs failure mode). Fresh repos get the community
README (+configs when audio); existing repos get only the configs key
ensured; metadata.jsonl rebuilt to cover ALL audio in the repo
(titles = stems, descriptions from local moves via opts.localMoves,
existing rows kept). Rename-cleanup-doomed files excluded (rule 4).
fetchRepoText distinguishes 404 from network failure: on failure the
share proceeds WITHOUT touching README/metadata (never clobber).
- compressAndRepublish: README(ensured) + metadata.jsonl ride the same
commit, computed against post-compress state (.ogg in, stale .wav out)
- folder.js passes the folder's moves as the metadata description source

Startup auto-repair (deliverable B):
- HF.repairOwnedViewerConfigs: own-namespace only, skip READMEs already
carrying configs, skip no-audio repos, one commit per repo, per-repo
try/catch, {checked,fixed,errors}
- main.js runViewerConfigMigration: runs automatically post-connect,
gated by IndexedDB kv 'viewer_config_migration_v1' so it's once per
browser profile — flag only written on a zero-error sweep (flaky runs
retry next session); bump the key to force a global re-sweep
- listSharedMoves projection gains description/sharedStem/fromCommunity

Compatibility: no move/audio file renamed or rewritten anywhere; import
pairing and countMoves ignore metadata.jsonl (endsWith '.json' filters).
35 new tests (viewer helpers, uploadMove viewer paths, sweep guards);
96/96 green.

docs/hf-viewer-fix-spec.md ADDED
@@ -0,0 +1,169 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Spec: make shared datasets playable in the HF dataset-viewer
2
+
3
+ **Goal:** every dataset Marionette publishes (new shares AND already-published ones
4
+ owned by the signed-in user) gets a working Hugging Face dataset-viewer page with a
5
+ playable audio column — without changing the on-Hub data layout in any way.
6
+
7
+ Status: root cause diagnosed and fix **verified live** on 2026-07-03. Two repos were
8
+ fixed manually as proof:
9
+
10
+ - Test bed: `RemiFabre/marionette-viewer-test` (viewer works; safe to delete)
11
+ - Production: `pollen-robotics/reachy-mini-emotions-library` (81 rows, playable
12
+ audio + title + description columns). Look at its `README.md` and
13
+ `metadata.jsonl` on the Hub — they are the reference output of this spec.
14
+
15
+ This spec describes app-side work in `lib/hf.js` / `main.js` so the fix applies
16
+ automatically. Nothing here changes the move file format.
17
+
18
+ ---
19
+
20
+ ## 1. Root cause (why the viewer is broken today)
21
+
22
+ The viewer picks a loader by **file-extension inference** when the README has no
23
+ `configs:` YAML. Our repos contain `<stem>.json` motion files, so it picks the
24
+ JSON builder, which chokes on the motion format:
25
+
26
+ ```
27
+ pyarrow.lib.ArrowInvalid: JSON parse error: Column() changed from object to number
28
+ ```
29
+
30
+ One failed config disables the **entire** viewer, audio previews included
31
+ (`is-valid` returns all-false, e.g. `Anne-Charlotte/reachy-songs`).
32
+
33
+ ## 2. The fix (what a "viewer-ready" repo looks like)
34
+
35
+ Two additive files; zero changes to motion/audio files:
36
+
37
+ **(a) README frontmatter gains a `configs:` key** pointing the viewer at the audio
38
+ files only. This forces the `audiofolder` builder and the motion JSONs are ignored:
39
+
40
+ ```yaml
41
+ configs:
42
+ - config_name: default
43
+ data_files:
44
+ - split: train
45
+ path:
46
+ - "data/*.ogg" # adjust to actual layout + extensions, see rules
47
+ - "data/metadata.jsonl"
48
+ ```
49
+
50
+ **(b) `metadata.jsonl`** (optional but required by this spec — without it the
51
+ viewer shows anonymous audio rows). One JSON line per audio file, living **in the
52
+ same directory as the audio**, `file_name` relative to that directory:
53
+
54
+ ```jsonl
55
+ {"file_name": "amazed1.ogg", "title": "amazed1", "description": "When you discover something extraordinary…", "motion_file": "amazed1.json"}
56
+ ```
57
+
58
+ Columns: `title` = move stem, `description` = the move JSON's `description` field,
59
+ `motion_file` = the sidecar's motion file. Do **not** name a column `label`
60
+ (reserved by audiofolder for class labels).
61
+
62
+ ### Hard rules (each one was hit or verified during the manual fix)
63
+
64
+ 1. **Every glob listed in `data_files` must match ≥ 1 file**, or the config errors
65
+ and the viewer stays broken. Therefore: emit one glob per audio extension
66
+ actually present in the repo (`.ogg` / `.opus` / `.wav`), and **omit the whole
67
+ `configs:` block when the repo has no audio at all**.
68
+ 2. Globs are relative to the repo root. Our repos come in two layouts: files under
69
+ `data/` (JS-app uploads) or at the repo root (e.g. the Python-seeded
70
+ emotions/dances libraries). Generate globs for the layout the repo actually has.
71
+ 3. The metadata file must be named **exactly `metadata.jsonl`** — NEVER
72
+ `metadata.json`: a `.json` name would be ingested as a move by BOTH loaders
73
+ (`reachy_mini` globs `*.json`; our importer filters `endsWith(".json")`).
74
+ `.jsonl` is provably invisible to both (verified against their source).
75
+ 4. Every `file_name` in `metadata.jsonl` must reference an audio file that exists
76
+ in the repo **after the commit** (mind `compressAndRepublish` deleting stale
77
+ `.wav` files in the same operation).
78
+ 5. Preserve everything already in the README — especially the
79
+ `reachy_mini_community_moves` tag (Community-list discovery depends on it) and
80
+ any hand-written body (the emotions library README is hand-maintained).
81
+ 6. `marionette.json` must never appear in `data_files`.
82
+
83
+ ## 3. Deliverable A — new & updated shares emit the fix
84
+
85
+ Touch points in `lib/hf.js`:
86
+
87
+ - **`buildCommunityReadme()` (~line 602)** — for fresh repos, embed the `configs:`
88
+ block in the generated frontmatter. It now needs to know which audio extensions
89
+ the repo will contain (pass this in from callers). No audio → no block.
90
+ - **New helper `ensureViewerConfig(readmeText, globs)`** — for repos that already
91
+ have a README: parse the frontmatter, insert or update **only** the `configs:`
92
+ key, and leave every other byte untouched. This is mandatory for the repair
93
+ sweep (Deliverable B) and recommended for `uploadMove` on existing repos —
94
+ full README regeneration would clobber customized READMEs like the emotions
95
+ library's (which already carries this fix; regenerating it would silently
96
+ remove the block — that is the current-code failure mode this spec eliminates).
97
+ - **`uploadMove()` (~line 381)** — in the same commit as the move, write/refresh
98
+ `data/metadata.jsonl` covering **all** moves in the repo, not just the new one.
99
+ Source for titles/descriptions: the local folder's moves in IndexedDB (labels,
100
+ descriptions, stems). Mind stem rules: local shares use `slugify(move.label)`,
101
+ imported moves keep their upstream stem (see the `m.id.split("/").pop()` logic
102
+ in `compressAndRepublish`, ~line 487). For repo files with no local
103
+ counterpart, keep their existing metadata row if present, else emit a row with
104
+ `title` = stem and no description.
105
+ - **`compressAndRepublish()` (~line 475)** — regenerate README (via
106
+ `ensureViewerConfig`) and `metadata.jsonl` inside its single commit. Audio is
107
+ forced to Opus here, so post-commit extensions are `.ogg`/`.opus` — rule 4.
108
+ - `publishFolderTuning()` — no change needed.
109
+
110
+ ## 4. Deliverable B — startup auto-repair of already-published datasets
111
+
112
+ Copy the existing, proven pattern: `HF.repairOwnedCommunityTags()`
113
+ (`lib/hf.js:325-379`) driven by `runCommunityTagMigration()` (`main.js:437-453`) —
114
+ once per session when a token is available, dedup shared moves to unique repos,
115
+ skip repos not in the user's namespace, per-repo try/catch, `{checked, fixed,
116
+ errors}` telemetry, success toast.
117
+
118
+ New sweep (parallel function, or extend the existing one):
119
+
120
+ For each unique owned repo behind locally-shared moves:
121
+
122
+ 1. Fetch `README.md` (`resolve/main/README.md`). If its frontmatter already has a
123
+ `configs:` key → **skip** (idempotent; never clobber).
124
+ 2. List the repo files (`listDatasetFiles`, ~line 88). No audio files → skip.
125
+ 3. Build the extension-accurate globs + `metadata.jsonl` (rules above; local
126
+ IndexedDB is the description source; fall back to omitting descriptions for
127
+ moves not known locally).
128
+ 4. Push `README.md` (via `ensureViewerConfig`) + `metadata.jsonl` in **one**
129
+ commit, e.g. `"Marionette: enable dataset-viewer audio previews"`.
130
+
131
+ Scope note: keep the strict own-namespace check. Org repos (e.g.
132
+ `pollen-robotics/*`) are out of scope for the sweep — the emotions library is
133
+ already fixed manually; `pollen-robotics/reachy-mini-dances-library` still has
134
+ the broken viewer and can get the same manual one-commit treatment.
135
+
136
+ ## 5. Compatibility invariants (the "0% breakage" contract)
137
+
138
+ - On-Hub data layout unchanged: no move `.json`/audio file is renamed, moved, or
139
+ rewritten by this feature. Only `README.md` + `metadata.jsonl` are written.
140
+ - `reachy_mini` (`src/reachy_mini/motion/recorded_move.py`, GitHub
141
+ `pollen-robotics/reachy_mini`): `RecordedMoves.process()` globs `*.json` and
142
+ `data/*.json`, pairs audio by stem — sees no difference.
143
+ - Marionette JS importer (`importDataset`, `lib/hf.js:121-276`): ingests
144
+ `endsWith(".json")` except `marionette.json`, pairs audio by stem — sees no
145
+ difference. Same for `countMoves`.
146
+ - Python Marionette app publishes the same format; the YAML `configs:` key is
147
+ inert metadata to it. (If it also regenerates READMEs on share, it will need
148
+ the same `ensureViewerConfig` treatment eventually — out of scope here, but
149
+ don't make the JS format diverge.)
150
+
151
+ ## 6. Verification (required before calling this done)
152
+
153
+ 1. Unit tests: `ensureViewerConfig` is idempotent, preserves an arbitrary README
154
+ body byte-for-byte outside the `configs:` key, handles no-audio (emits
155
+ nothing / removes nothing), multi-extension repos, root vs `data/` layouts.
156
+ 2. Live end-to-end on a scratch dataset under the user's namespace (pattern:
157
+ `RemiFabre/marionette-viewer-test`): share a move from the app, then poll —
158
+
159
+ ```bash
160
+ curl -s "https://datasets-server.huggingface.co/is-valid?dataset=<repo>" # want viewer:true (takes 1–4 min)
161
+ curl -s "https://datasets-server.huggingface.co/splits?dataset=<repo>" # want failed: []
162
+ curl -s "https://datasets-server.huggingface.co/first-rows?dataset=<repo>&config=default&split=train"
163
+ # want features [audio:Audio, title, description, motion_file] and playable row assets
164
+ ```
165
+
166
+ 3. Re-import the scratch dataset through the app's Community flow: move count and
167
+ audio pairing identical to pre-change.
168
+ 4. For the repair path, byte-diff the README before/after on a repo with a
169
+ customized body: the only difference must be the `configs:` block.
lib/db.js CHANGED
@@ -137,6 +137,12 @@ export async function listSharedMoves() {
137
  label: m.label,
138
  shared: m.shared,
139
  sharedUrl: m.sharedUrl,
 
 
 
 
 
 
140
  }));
141
  }
142
 
 
137
  label: m.label,
138
  shared: m.shared,
139
  sharedUrl: m.sharedUrl,
140
+ // Used by the viewer-config repair sweep: description feeds the
141
+ // metadata.jsonl rows, sharedStem/fromCommunity resolve which
142
+ // on-Hub stem the move publishes under. Still no blobs.
143
+ description: m.description,
144
+ sharedStem: m.sharedStem,
145
+ fromCommunity: m.fromCommunity,
146
  }));
147
  }
148
 
lib/hf.js CHANGED
@@ -109,6 +109,21 @@ export const HF = {
109
  return res;
110
  },
111
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
112
  async countMoves(repoId, token) {
113
  // Cheap-ish probe: list files and count .json entries. Used by the
114
  // community list so each row can show the real number before import.
@@ -378,6 +393,71 @@ export const HF = {
378
  return { checked, fixed, errors };
379
  },
380
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
381
  async uploadMove(move, folderName, token, opts = {}) {
382
  if (!token) throw new Error("Sign in to Hugging Face to share.");
383
  const username = await this.getHfNamespace(token);
@@ -428,17 +508,6 @@ export const HF = {
428
  path: `data/${moveStem}.json`,
429
  content: new Blob([JSON.stringify(moveJson, null, 2)], { type: "application/json" }),
430
  },
431
- // The community-tag README is REQUIRED for the dataset to
432
- // appear in the Marionette "Community" tab. Without this
433
- // file, HF doesn't carry the `reachy_mini_community_moves`
434
- // tag, and our list endpoint (?filter=…) excludes the
435
- // dataset. Existing Python Marionette uses the same tag
436
- // for cross-app discovery. Re-uploaded on every share to
437
- // keep the frontmatter current if it ever drifts.
438
- {
439
- path: "README.md",
440
- content: new Blob([buildCommunityReadme(folderName, repoId)], { type: "text/markdown" }),
441
- },
442
  ];
443
  if (move.audioBlob) {
444
  // Name the sidecar by its real container so the daemon's loader
@@ -457,6 +526,81 @@ export const HF = {
457
  });
458
  }
459
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
460
  await hfUploadFiles({
461
  repo: { type: "dataset", name: repoId },
462
  credentials,
@@ -470,9 +614,8 @@ export const HF = {
470
  // files still carry the old slug; drop them so community import
471
  // doesn't show the move twice. A failed delete only leaves an orphan
472
  // (a harmless duplicate), never data loss.
473
- const oldStem = move.sharedStem;
474
  if (oldStem && oldStem !== moveStem) {
475
- if ((opts.protectedStems || []).includes(oldStem)) {
476
  // Slugs are slugify(label) with no uniqueness: another move
477
  // in this folder currently publishes under the old slug, so
478
  // `data/<oldStem>.*` is that move's LIVE copy — deleting it
@@ -565,21 +708,64 @@ export const HF = {
565
  m.frameCount = moveJson.set_target_data.length;
566
  await putMove(m);
567
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
568
  await hfUploadFiles({
569
  repo: { type: "dataset", name: repoId },
570
  credentials,
571
  files,
572
  commitTitle: "Compress: 50 Hz motion + Opus audio",
573
  });
574
- // Delete only the .wav sidecars we've just replaced with .ogg — never
575
- // a .wav whose move we didn't re-upload (that would orphan its audio).
576
- const upstream = await this.listDatasetFiles(repoId, token);
577
- const staleWavs = upstream
578
- .map((f) => f.path)
579
- .filter((p) => {
580
- const mm = p.match(/^(?:data\/)?(.+)\.wav$/i);
581
- return mm && audioStems.has(mm[1]);
582
- });
583
  if (staleWavs.length) {
584
  await hfDeleteFiles({
585
  repo: { type: "dataset", name: repoId },
@@ -630,6 +816,178 @@ export const HF = {
630
  },
631
  };
632
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
633
  // Build the optional `marionette.json` payload, OR return null when
634
  // the folder has no non-default settings worth recording. The schema
635
  // version exists so we can add fields later without ambiguity.
@@ -645,9 +1003,12 @@ function buildMarionetteMeta({ audioOffsetMs } = {}) {
645
  // README that carries the community discovery tag in YAML
646
  // frontmatter. Mirrors the format the Python Marionette app uses, so
647
  // datasets published from either app are indistinguishable to the
648
- // listing endpoint.
649
- function buildCommunityReadme(folderName, repoId) {
 
 
650
  const pretty = (folderName || "marionette").replace(/[-_]+/g, " ");
 
651
  return `---
652
  license: apache-2.0
653
  task_categories:
@@ -657,7 +1018,7 @@ language:
657
  tags:
658
  - ${HF_COMMUNITY_TAG}
659
  pretty_name: ${pretty} • Reachy Mini Moves
660
- ---
661
 
662
  # ${pretty} • Reachy Mini Moves
663
 
 
109
  return res;
110
  },
111
 
112
+ // Fetch a small text file from a repo, distinguishing "definitely
113
+ // absent" from "couldn't tell". Returns the text, or null on a clean
114
+ // 404, and THROWS on anything else (network flake, 5xx) — callers
115
+ // that decide "no README → generate a fresh one" must not mistake a
116
+ // transient failure for absence and clobber a customized file.
117
+ async fetchRepoText(repoId, path, token) {
118
+ try {
119
+ const res = await this.downloadFile(repoId, path, token);
120
+ return await res.text();
121
+ } catch (e) {
122
+ if (/HF 404/.test(String(e?.message || e))) return null;
123
+ throw e;
124
+ }
125
+ },
126
+
127
  async countMoves(repoId, token) {
128
  // Cheap-ish probe: list files and count .json entries. Used by the
129
  // community list so each row can show the real number before import.
 
393
  return { checked, fixed, errors };
394
  },
395
 
396
+ // Startup auto-repair (docs/hf-viewer-fix-spec.md, deliverable B):
397
+ // make every dataset the user owns viewer-ready. For each unique
398
+ // owned repo behind their locally-shared moves / linked folders:
399
+ // skip if the README already has a `configs:` key (idempotent, never
400
+ // clobber), skip if the repo has no audio, otherwise push README
401
+ // (via ensureViewerConfig — byte-preserving) + a rebuilt
402
+ // metadata.jsonl in ONE commit. Same shape as
403
+ // repairOwnedCommunityTags: strict own-namespace check, per-repo
404
+ // try/catch, { checked, fixed, errors } telemetry.
405
+ //
406
+ // `repos`: Map(repoId → localMoves[]) — the local IndexedDB moves
407
+ // known to live in each repo, used as the title/description source.
408
+ async repairOwnedViewerConfigs({ token, repos, onProgress }) {
409
+ if (!token || !repos?.size) return { checked: 0, fixed: 0, errors: 0 };
410
+
411
+ let myNamespace;
412
+ try {
413
+ myNamespace = await this.getHfNamespace(token);
414
+ } catch (e) {
415
+ Log.warn("repairOwnedViewerConfigs: namespace lookup failed", { error: String(e) });
416
+ return { checked: 0, fixed: 0, errors: 1 };
417
+ }
418
+
419
+ const credentials = { accessToken: token };
420
+ let checked = 0, fixed = 0, errors = 0;
421
+ for (const [repoId, localMoves] of repos) {
422
+ if (repoId.split("/")[0] !== myNamespace) continue; // not ours
423
+ checked++;
424
+ try {
425
+ const existingReadme = await this.fetchRepoText(repoId, "README.md", token);
426
+ if (existingReadme !== null && hasViewerConfig(existingReadme)) continue; // already fixed
427
+
428
+ const paths = (await this.listDatasetFiles(repoId, token)).map((f) => f.path);
429
+ const plan = viewerConfigPlan(paths);
430
+ if (!plan) continue; // no audio → a config would only break the viewer
431
+
432
+ const readme = existingReadme === null
433
+ ? buildCommunityReadme(repoId.split("/").pop(), repoId, plan.dataFiles)
434
+ : ensureViewerConfig(existingReadme, plan.dataFiles);
435
+ const existingRows = parseMetadataJsonl(
436
+ paths.includes(plan.metadataPath)
437
+ ? await this.fetchRepoText(repoId, plan.metadataPath, token)
438
+ : null,
439
+ );
440
+ const rows = buildViewerMetadataRows({ paths, plan, localMoves, existingRows });
441
+
442
+ await hfUploadFiles({
443
+ repo: { type: "dataset", name: repoId },
444
+ credentials,
445
+ files: [
446
+ { path: "README.md", content: new Blob([readme], { type: "text/markdown" }) },
447
+ { path: plan.metadataPath, content: new Blob([buildMetadataJsonl(rows)], { type: "application/json" }) },
448
+ ],
449
+ commitTitle: "Marionette: enable dataset-viewer audio previews",
450
+ });
451
+ fixed++;
452
+ onProgress?.(fixed, repos.size);
453
+ } catch (e) {
454
+ errors++;
455
+ Log.warn("repairOwnedViewerConfigs: per-repo failure", { repoId, error: String(e) });
456
+ }
457
+ }
458
+ return { checked, fixed, errors };
459
+ },
460
+
461
  async uploadMove(move, folderName, token, opts = {}) {
462
  if (!token) throw new Error("Sign in to Hugging Face to share.");
463
  const username = await this.getHfNamespace(token);
 
508
  path: `data/${moveStem}.json`,
509
  content: new Blob([JSON.stringify(moveJson, null, 2)], { type: "application/json" }),
510
  },
 
 
 
 
 
 
 
 
 
 
 
511
  ];
512
  if (move.audioBlob) {
513
  // Name the sidecar by its real container so the daemon's loader
 
526
  });
527
  }
528
 
529
+ // README + dataset-viewer support, in the SAME commit as the move
530
+ // (docs/hf-viewer-fix-spec.md):
531
+ // - fresh repos get the generated community README — it carries
532
+ // the reachy_mini_community_moves tag the Community tab's
533
+ // listing filter depends on, plus the viewer `configs:` block
534
+ // when the repo will contain audio;
535
+ // - existing repos get ONLY their `configs:` key inserted or
536
+ // updated (ensureViewerConfig) — never a wholesale rewrite,
537
+ // which used to clobber customized READMEs;
538
+ // - metadata.jsonl is refreshed to cover ALL audio in the repo
539
+ // (titles from stems, descriptions from local IndexedDB moves
540
+ // via opts.localMoves, existing rows kept for files we don't
541
+ // know locally).
542
+ // Any failure here is logged and skipped — viewer housekeeping
543
+ // must never block a share; the startup repair sweep finishes the
544
+ // job next session.
545
+ const oldStem = move.sharedStem;
546
+ const cleanupPlanned = !!(oldStem && oldStem !== moveStem
547
+ && !(opts.protectedStems || []).includes(oldStem));
548
+ try {
549
+ const upstream = await this.listDatasetFiles(repoId, token);
550
+ // Predicted post-share repo content: what's upstream, minus the
551
+ // old-stem files the rename cleanup below is about to delete,
552
+ // plus everything in this commit (rule 4: metadata/globs must
553
+ // only reference files that exist once the dust settles).
554
+ const staleNames = cleanupPlanned
555
+ ? new Set([`${oldStem}.json`, ...SOUND_EXTENSIONS.map((ext) => `${oldStem}${ext}`)])
556
+ : new Set();
557
+ const predicted = new Set(
558
+ upstream.map((f) => f.path).filter((p) => !staleNames.has(p.split("/").pop())),
559
+ );
560
+ for (const f of files) predicted.add(f.path);
561
+ const plan = viewerConfigPlan([...predicted]);
562
+
563
+ const existingReadme = await this.fetchRepoText(repoId, "README.md", token);
564
+ if (existingReadme === null) {
565
+ files.push({
566
+ path: "README.md",
567
+ content: new Blob([buildCommunityReadme(folderName, repoId, plan?.dataFiles)], { type: "text/markdown" }),
568
+ });
569
+ } else if (plan) {
570
+ const ensured = ensureViewerConfig(existingReadme, plan.dataFiles);
571
+ if (ensured !== existingReadme) {
572
+ files.push({ path: "README.md", content: new Blob([ensured], { type: "text/markdown" }) });
573
+ }
574
+ }
575
+
576
+ if (plan) {
577
+ const existingRows = parseMetadataJsonl(
578
+ predicted.has(plan.metadataPath)
579
+ ? await this.fetchRepoText(repoId, plan.metadataPath, token)
580
+ : null,
581
+ );
582
+ // The move being shared goes LAST so its entry (under the
583
+ // stem it's being published as right now) wins over any
584
+ // stale sibling data.
585
+ const localMoves = [
586
+ ...(opts.localMoves || []),
587
+ { ...move, sharedStem: moveStem, fromCommunity: false },
588
+ ];
589
+ const rows = buildViewerMetadataRows({ paths: [...predicted], plan, localMoves, existingRows });
590
+ files.push({
591
+ path: plan.metadataPath,
592
+ content: new Blob([buildMetadataJsonl(rows)], { type: "application/json" }),
593
+ });
594
+ }
595
+ } catch (e) {
596
+ // Couldn't determine the repo's state (network flake, 5xx).
597
+ // Ship the move without touching README/metadata: a fresh repo
598
+ // missing its community tag self-heals via the tag-repair sweep
599
+ // next session, and an existing README is left alone rather
600
+ // than risk clobbering a customized one.
601
+ Log.warn("uploadMove: viewer/README housekeeping skipped", { repoId, error: String(e) });
602
+ }
603
+
604
  await hfUploadFiles({
605
  repo: { type: "dataset", name: repoId },
606
  credentials,
 
614
  // files still carry the old slug; drop them so community import
615
  // doesn't show the move twice. A failed delete only leaves an orphan
616
  // (a harmless duplicate), never data loss.
 
617
  if (oldStem && oldStem !== moveStem) {
618
+ if (!cleanupPlanned) {
619
  // Slugs are slugify(label) with no uniqueness: another move
620
  // in this folder currently publishes under the old slug, so
621
  // `data/<oldStem>.*` is that move's LIVE copy — deleting it
 
708
  m.frameCount = moveJson.set_target_data.length;
709
  await putMove(m);
710
  }
711
+ // Stale sidecars to drop once the new files are committed: only
712
+ // the .wav files we're replacing with .ogg in this very commit —
713
+ // never a .wav whose move we didn't re-upload (that would orphan
714
+ // its audio). Listed BEFORE the upload so the same snapshot also
715
+ // feeds the viewer config below.
716
+ const upstream = await this.listDatasetFiles(repoId, token);
717
+ const upstreamPaths = upstream.map((f) => f.path);
718
+ const staleWavs = upstreamPaths.filter((p) => {
719
+ const mm = p.match(/^(?:data\/)?(.+)\.wav$/i);
720
+ return mm && audioStems.has(mm[1]);
721
+ });
722
+
723
+ // Viewer support (docs/hf-viewer-fix-spec.md): refresh the README
724
+ // `configs:` key + metadata.jsonl inside the same commit, computed
725
+ // against the post-compress state (new .ogg files in, stale .wav
726
+ // out — rule 4). These repos are owned and often carry customized
727
+ // READMEs, so only ensureViewerConfig ever touches them. Failures
728
+ // are logged and skipped: compression must not be blocked.
729
+ try {
730
+ const stale = new Set(staleWavs);
731
+ const predicted = new Set(upstreamPaths.filter((p) => !stale.has(p)));
732
+ for (const f of files) predicted.add(f.path);
733
+ const plan = viewerConfigPlan([...predicted]);
734
+ if (plan) {
735
+ const existingReadme = await this.fetchRepoText(repoId, "README.md", token);
736
+ const readme = existingReadme === null
737
+ ? buildCommunityReadme(repoId.split("/").pop(), repoId, plan.dataFiles)
738
+ : ensureViewerConfig(existingReadme, plan.dataFiles);
739
+ if (readme !== existingReadme) {
740
+ files.push({ path: "README.md", content: new Blob([readme], { type: "text/markdown" }) });
741
+ }
742
+ const existingRows = parseMetadataJsonl(
743
+ predicted.has(plan.metadataPath)
744
+ ? await this.fetchRepoText(repoId, plan.metadataPath, token)
745
+ : null,
746
+ );
747
+ // Metadata rows must map by the stems used in THIS commit:
748
+ // compress publishes local moves under slugify(label) (not
749
+ // sharedStem, which may lag behind a local rename).
750
+ const localMoves = moves.map((m) => (m.fromCommunity ? m : { ...m, sharedStem: slugify(m.label) }));
751
+ const rows = buildViewerMetadataRows({
752
+ paths: [...predicted], plan, localMoves, existingRows,
753
+ });
754
+ files.push({
755
+ path: plan.metadataPath,
756
+ content: new Blob([buildMetadataJsonl(rows)], { type: "application/json" }),
757
+ });
758
+ }
759
+ } catch (e) {
760
+ Log.warn("compressAndRepublish: viewer housekeeping skipped", { repoId, error: String(e) });
761
+ }
762
+
763
  await hfUploadFiles({
764
  repo: { type: "dataset", name: repoId },
765
  credentials,
766
  files,
767
  commitTitle: "Compress: 50 Hz motion + Opus audio",
768
  });
 
 
 
 
 
 
 
 
 
769
  if (staleWavs.length) {
770
  await hfDeleteFiles({
771
  repo: { type: "dataset", name: repoId },
 
816
  },
817
  };
818
 
819
+ // ─── HF dataset-viewer support (docs/hf-viewer-fix-spec.md) ─────────
820
+ //
821
+ // Without a `configs:` key in the README frontmatter, the dataset-viewer
822
+ // infers a loader from file extensions, picks the JSON builder for our
823
+ // motion files, chokes on their shape, and the WHOLE viewer (audio
824
+ // previews included) stays broken. The fix is two additive files —
825
+ // README `configs:` pointing the viewer at audio only, plus a
826
+ // `metadata.jsonl` giving each audio row a title/description — and zero
827
+ // changes to the move files themselves. Verified live on
828
+ // pollen-robotics/reachy-mini-emotions-library (2026-07-03).
829
+
830
+ // Decide what the viewer config should look like for a repo containing
831
+ // `paths` (post-commit). Returns null when the repo has no audio at all
832
+ // (a config whose globs match nothing errors out and re-breaks the
833
+ // viewer — rule 1 of the spec). Repos come in two layouts (files under
834
+ // data/ for JS-app uploads, at the root for Python-seeded libraries);
835
+ // globs are emitted for the layout the audio actually uses, preferring
836
+ // data/ in the (never-seen) mixed case so metadata.jsonl can sit next
837
+ // to the audio it describes.
838
+ export function viewerConfigPlan(paths) {
839
+ const extsByPrefix = { "data/": new Set(), "": new Set() };
840
+ for (const p of paths) {
841
+ const m = /^(data\/)?[^/]+(\.[a-z0-9]+)$/i.exec(p);
842
+ if (!m) continue;
843
+ const ext = m[2].toLowerCase();
844
+ if (!SOUND_EXTENSIONS.includes(ext)) continue;
845
+ extsByPrefix[m[1] || ""].add(ext);
846
+ }
847
+ const prefix = extsByPrefix["data/"].size ? "data/" : "";
848
+ const exts = extsByPrefix[prefix];
849
+ if (!exts.size) return null;
850
+ const metadataPath = `${prefix}metadata.jsonl`;
851
+ const dataFiles = [
852
+ ...[...exts].sort().map((e) => `${prefix}*${e}`),
853
+ metadataPath,
854
+ ];
855
+ return { prefix, dataFiles, metadataPath };
856
+ }
857
+
858
+ // Render the exact YAML block the viewer needs. Kept as a plain string
859
+ // template (no YAML lib): the shape is fixed and byte-stable output is
860
+ // what makes ensureViewerConfig idempotent.
861
+ export function renderViewerConfigBlock(dataFiles) {
862
+ return [
863
+ "configs:",
864
+ " - config_name: default",
865
+ " data_files:",
866
+ " - split: train",
867
+ " path:",
868
+ ...dataFiles.map((p) => ` - "${p}"`),
869
+ ].join("\n");
870
+ }
871
+
872
+ // Insert or update ONLY the top-level `configs:` key of a README's YAML
873
+ // frontmatter; every other byte (rest of the frontmatter, the whole
874
+ // body) is preserved verbatim. This is what lets us touch customized,
875
+ // hand-maintained READMEs (e.g. the emotions library) without
876
+ // clobbering them — regenerating the README wholesale is the failure
877
+ // mode this replaces. No audio (empty dataFiles) → README returned
878
+ // untouched.
879
+ export function ensureViewerConfig(readmeText, dataFiles) {
880
+ if (!dataFiles || !dataFiles.length) return readmeText;
881
+ const block = renderViewerConfigBlock(dataFiles);
882
+ const fmMatch = /^---\r?\n([\s\S]*?)\r?\n---(?=\r?\n|$)/.exec(readmeText);
883
+ if (!fmMatch) {
884
+ // No frontmatter at all: prepend one carrying just the config.
885
+ return `---\n${block}\n---\n\n${readmeText}`;
886
+ }
887
+ const fm = fmMatch[1];
888
+ const rest = readmeText.slice(fmMatch[0].length); // starts at the newline after the closing ---
889
+ const lines = fm.split("\n");
890
+ // Locate an existing top-level `configs:` key: from its line to the
891
+ // next top-level key (a line starting with a non-space, non-list,
892
+ // non-comment character) or the end of the frontmatter.
893
+ let start = -1;
894
+ let end = lines.length;
895
+ for (let i = 0; i < lines.length; i++) {
896
+ if (start === -1) {
897
+ if (/^configs:(\s|$)/.test(lines[i])) start = i;
898
+ } else if (/^[^\s#-]/.test(lines[i])) {
899
+ end = i;
900
+ break;
901
+ }
902
+ }
903
+ const newFm = start !== -1
904
+ ? [...lines.slice(0, start), block, ...lines.slice(end)].join("\n")
905
+ : `${fm}\n${block}`;
906
+ return `---\n${newFm}\n---${rest}`;
907
+ }
908
+
909
+ // True when a README's frontmatter already carries a top-level
910
+ // `configs:` key — the repair sweep's "already fixed, never clobber"
911
+ // check.
912
+ export function hasViewerConfig(readmeText) {
913
+ const fmMatch = /^---\r?\n([\s\S]*?)\r?\n---(?=\r?\n|$)/.exec(readmeText || "");
914
+ if (!fmMatch) return false;
915
+ return fmMatch[1].split("\n").some((l) => /^configs:(\s|$)/.test(l));
916
+ }
917
+
918
+ // Parse an existing metadata.jsonl into Map(file_name → row) so repair /
919
+ // re-share can keep rows for audio files we know nothing about locally.
920
+ export function parseMetadataJsonl(text) {
921
+ const map = new Map();
922
+ if (!text) return map;
923
+ for (const line of text.split("\n")) {
924
+ const t = line.trim();
925
+ if (!t) continue;
926
+ try {
927
+ const row = JSON.parse(t);
928
+ if (row && typeof row.file_name === "string") map.set(row.file_name, row);
929
+ } catch { /* skip malformed lines, never fail the whole file */ }
930
+ }
931
+ return map;
932
+ }
933
+
934
+ // One metadata row per audio file in the plan's directory. Sources, in
935
+ // priority order: the local IndexedDB move (title = stem, description =
936
+ // what the published move JSON carries), the repo's existing
937
+ // metadata.jsonl row, else title-only. Column names are fixed by the
938
+ // audiofolder loader — and `label` is reserved by it for class labels,
939
+ // so we never emit one.
940
+ export function buildViewerMetadataRows({ paths, plan, localMoves = [], existingRows = new Map() }) {
941
+ const { prefix } = plan;
942
+ const inDir = (p) => (prefix ? p.startsWith(prefix) && !p.slice(prefix.length).includes("/") : !p.includes("/"));
943
+ const base = (p) => (prefix ? p.slice(prefix.length) : p);
944
+
945
+ const jsonStems = new Set();
946
+ const audio = [];
947
+ for (const p of paths) {
948
+ if (!inDir(p)) continue;
949
+ const b = base(p);
950
+ if (b === "marionette.json" || b === "metadata.jsonl") continue;
951
+ if (b.endsWith(".json")) { jsonStems.add(b.slice(0, -".json".length)); continue; }
952
+ const ext = /(\.[a-z0-9]+)$/i.exec(b)?.[1]?.toLowerCase();
953
+ if (ext && SOUND_EXTENSIONS.includes(ext)) audio.push({ fileName: b, stem: b.slice(0, -ext.length) });
954
+ }
955
+
956
+ // Published stem of a local move: imported moves keep their upstream
957
+ // stem (tail of "datasetId/stem" ids), local shares use the slug they
958
+ // were last published under (sharedStem, falling back to the label).
959
+ const byStem = new Map();
960
+ for (const m of localMoves) {
961
+ const stem = m?.fromCommunity && m.id?.includes("/")
962
+ ? m.id.split("/").pop()
963
+ : (m?.sharedStem || slugify(m?.label || ""));
964
+ if (stem) byStem.set(stem, m);
965
+ }
966
+
967
+ audio.sort((a, b) => a.fileName.localeCompare(b.fileName));
968
+ return audio.map(({ fileName, stem }) => {
969
+ const local = byStem.get(stem);
970
+ const prev = existingRows.get(fileName);
971
+ const description = local
972
+ ? (local.description || local.label || "")
973
+ : (prev?.description || "");
974
+ return {
975
+ file_name: fileName,
976
+ title: stem,
977
+ ...(description ? { description } : {}),
978
+ ...(jsonStems.has(stem) ? { motion_file: `${stem}.json` } : {}),
979
+ };
980
+ });
981
+ }
982
+
983
+ // Serialize rows to the on-Hub metadata.jsonl format (one JSON object
984
+ // per line). MUST be named metadata.jsonl, never .json — a .json name
985
+ // would be ingested as a move by both the reachy_mini loader (globs
986
+ // *.json) and our own importer (endsWith ".json").
987
+ export function buildMetadataJsonl(rows) {
988
+ return rows.length ? rows.map((r) => JSON.stringify(r)).join("\n") + "\n" : "";
989
+ }
990
+
991
  // Build the optional `marionette.json` payload, OR return null when
992
  // the folder has no non-default settings worth recording. The schema
993
  // version exists so we can add fields later without ambiguity.
 
1003
  // README that carries the community discovery tag in YAML
1004
  // frontmatter. Mirrors the format the Python Marionette app uses, so
1005
  // datasets published from either app are indistinguishable to the
1006
+ // listing endpoint. `dataFiles` (from viewerConfigPlan) additionally
1007
+ // embeds the dataset-viewer `configs:` block — omitted when the repo
1008
+ // has no audio (an unmatched glob breaks the whole viewer).
1009
+ function buildCommunityReadme(folderName, repoId, dataFiles = null) {
1010
  const pretty = (folderName || "marionette").replace(/[-_]+/g, " ");
1011
+ const configs = dataFiles?.length ? `${renderViewerConfigBlock(dataFiles)}\n` : "";
1012
  return `---
1013
  license: apache-2.0
1014
  task_categories:
 
1018
  tags:
1019
  - ${HF_COMMUNITY_TAG}
1020
  pretty_name: ${pretty} • Reachy Mini Moves
1021
+ ${configs}---
1022
 
1023
  # ${pretty} • Reachy Mini Moves
1024
 
main.js CHANGED
@@ -38,7 +38,7 @@ import { connectToHost } from "https://cdn.jsdelivr.net/npm/@pollen-robotics/rea
38
  import { state, setRenderImpl } from "./lib/state.js";
39
  import { Log } from "./lib/log.js";
40
  import { busy, runCleanups } from "./lib/busy.js";
41
- import { ensureDefaultDataset, listSharedMoves, countAllMoves } from "./lib/db.js";
42
  import { HF } from "./lib/hf.js";
43
  import { runMigrationWithOverlay, runMigrationIfNeeded } from "./lib/audio-migrate.js";
44
  import { createMotorSubsystem, flat16 } from "./lib/motor.js";
@@ -366,6 +366,14 @@ async function bootEmbed() {
366
  Log.warn("community tag migration failed", { error: String(e) }),
367
  );
368
 
 
 
 
 
 
 
 
 
369
  // Audio re-upload retry: any shared moves that needed re-upload
370
  // during bootstrap get flushed now that we have a token. Background;
371
  // no overlay.
@@ -504,6 +512,56 @@ async function runStartupGreeting(robot, Audio_, Playback) {
504
  // current tags; if we own it and the tag is missing, push a README
505
  // to fix it. We don't touch already-tagged repos (don't clobber
506
  // user-customised READMEs).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
507
  let _tagMigrationRan = false;
508
  async function runCommunityTagMigration(robot) {
509
  if (_tagMigrationRan) return;
 
38
  import { state, setRenderImpl } from "./lib/state.js";
39
  import { Log } from "./lib/log.js";
40
  import { busy, runCleanups } from "./lib/busy.js";
41
+ import { ensureDefaultDataset, listSharedMoves, countAllMoves, kvGet, kvSet } from "./lib/db.js";
42
  import { HF } from "./lib/hf.js";
43
  import { runMigrationWithOverlay, runMigrationIfNeeded } from "./lib/audio-migrate.js";
44
  import { createMotorSubsystem, flat16 } from "./lib/motor.js";
 
366
  Log.warn("community tag migration failed", { error: String(e) }),
367
  );
368
 
369
+ // One-shot dataset-viewer repair: make every owned dataset playable
370
+ // on its HF page (README `configs:` + metadata.jsonl). Persisted in
371
+ // IndexedDB so it doesn't re-run every session; new shares emit the
372
+ // fix themselves.
373
+ runViewerConfigMigration(robot).catch((e) =>
374
+ Log.warn("viewer config migration failed", { error: String(e) }),
375
+ );
376
+
377
  // Audio re-upload retry: any shared moves that needed re-upload
378
  // during bootstrap get flushed now that we have a token. Background;
379
  // no overlay.
 
512
  // current tags; if we own it and the tag is missing, push a README
513
  // to fix it. We don't touch already-tagged repos (don't clobber
514
  // user-customised READMEs).
515
+ // Dataset-viewer repair sweep (docs/hf-viewer-fix-spec.md): every repo
516
+ // the user owns gets the README `configs:` + metadata.jsonl treatment so
517
+ // its HF page shows playable audio rows. Runs automatically once we have
518
+ // a token, then records completion in IndexedDB (kv) so later sessions
519
+ // skip it entirely — the per-repo check is idempotent anyway (repos
520
+ // whose README already carries `configs:` are never touched), and every
521
+ // new/updated share emits the fix by itself. The flag is only written
522
+ // when the sweep finishes with zero errors, so a flaky run retries next
523
+ // session. Bump the kv key's version suffix to force a global re-sweep
524
+ // after a future format change.
525
+ const VIEWER_MIGRATION_KV = "viewer_config_migration_v1";
526
+ let _viewerMigrationRan = false;
527
+ async function runViewerConfigMigration(robot) {
528
+ if (_viewerMigrationRan) return;
529
+ _viewerMigrationRan = true;
530
+ if (!robot._token) return;
531
+ if (await kvGet(VIEWER_MIGRATION_KV)) return;
532
+
533
+ // Group the locally-shared moves by the repo they live in; they're
534
+ // the metadata title/description source. Owned-namespace filtering
535
+ // happens inside the sweep (needs the whoAmI namespace).
536
+ const moves = await listSharedMoves();
537
+ const repos = new Map();
538
+ for (const m of moves) {
539
+ const repoId = m.sharedUrl?.split("/datasets/")[1]?.split(/[?#]/)[0];
540
+ if (!repoId) continue;
541
+ if (!repos.has(repoId)) repos.set(repoId, []);
542
+ repos.get(repoId).push(m);
543
+ }
544
+ if (!repos.size) {
545
+ // Nothing shared from this browser — nothing to repair, and
546
+ // nothing to gain from re-checking every session.
547
+ await kvSet(VIEWER_MIGRATION_KV, { done: true, at: Date.now(), checked: 0, fixed: 0 });
548
+ return;
549
+ }
550
+
551
+ const { checked, fixed, errors } = await HF.repairOwnedViewerConfigs({
552
+ token: robot._token,
553
+ repos,
554
+ });
555
+ Log.info("viewer config migration", { checked, fixed, errors });
556
+ if (fixed > 0) {
557
+ const s = fixed === 1 ? "" : "s";
558
+ toast(`Enabled HF audio previews on ${fixed} dataset${s}`, "success");
559
+ }
560
+ if (errors === 0) {
561
+ await kvSet(VIEWER_MIGRATION_KV, { done: true, at: Date.now(), checked, fixed });
562
+ }
563
+ }
564
+
565
  let _tagMigrationRan = false;
566
  async function runCommunityTagMigration(robot) {
567
  if (_tagMigrationRan) return;
tests/unit/hf-upload.test.js CHANGED
@@ -36,6 +36,7 @@ function makeMove(overrides = {}) {
36
 
37
  const origNamespace = HF.getHfNamespace;
38
  const origList = HF.listDatasetFiles;
 
39
 
40
  beforeEach(() => {
41
  resetHubStub();
@@ -46,11 +47,15 @@ beforeEach(() => {
46
  { path: "data/dance.ogg", _prefix: "data" },
47
  { path: "data/other.json", _prefix: "data" },
48
  ];
 
 
 
49
  });
50
 
51
  afterEach(() => {
52
  HF.getHfNamespace = origNamespace;
53
  HF.listDatasetFiles = origList;
 
54
  });
55
 
56
  describe("uploadMove rename cleanup", () => {
@@ -128,3 +133,110 @@ describe("uploadMove rename cleanup", () => {
128
  expect(paths).toContain("data/dance-2.ogg");
129
  });
130
  });
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
36
 
37
  const origNamespace = HF.getHfNamespace;
38
  const origList = HF.listDatasetFiles;
39
+ const origFetchText = HF.fetchRepoText;
40
 
41
  beforeEach(() => {
42
  resetHubStub();
 
47
  { path: "data/dance.ogg", _prefix: "data" },
48
  { path: "data/other.json", _prefix: "data" },
49
  ];
50
+ // No README / metadata upstream by default (fresh-repo behavior);
51
+ // also keeps unit tests off the real network.
52
+ HF.fetchRepoText = async () => null;
53
  });
54
 
55
  afterEach(() => {
56
  HF.getHfNamespace = origNamespace;
57
  HF.listDatasetFiles = origList;
58
+ HF.fetchRepoText = origFetchText;
59
  });
60
 
61
  describe("uploadMove rename cleanup", () => {
 
133
  expect(paths).toContain("data/dance-2.ogg");
134
  });
135
  });
136
+
137
+ describe("uploadMove viewer support", () => {
138
+ const fileText = async (call, path) => {
139
+ const f = call.arg.files.find((x) => x.path === path);
140
+ return f ? f.content.text() : null;
141
+ };
142
+
143
+ it("fresh repo: README carries the community tag AND the configs block; metadata covers all audio", async () => {
144
+ const move = makeMove();
145
+ await HF.uploadMove(move, "My Folder", "tok", { localMoves: [] });
146
+
147
+ const up = calls.find((c) => c.fn === "uploadFiles");
148
+ const readme = await fileText(up, "README.md");
149
+ expect(readme).toContain("reachy_mini_community_moves");
150
+ expect(readme).toContain("configs:");
151
+ expect(readme).toContain(`- "data/*.ogg"`);
152
+ expect(readme).toContain(`- "data/metadata.jsonl"`);
153
+
154
+ const meta = await fileText(up, "data/metadata.jsonl");
155
+ // Rename cleanup will delete data/dance.* right after this commit
156
+ // — the metadata must NOT reference them (rule 4), only the new
157
+ // stem's audio.
158
+ expect(meta).toContain(`"file_name":"dance-2.ogg"`);
159
+ expect(meta).not.toContain(`"file_name":"dance.ogg"`);
160
+ expect(meta).toContain(`"motion_file":"dance-2.json"`);
161
+ });
162
+
163
+ it("existing README: only the configs key is inserted, body preserved", async () => {
164
+ const custom = `---
165
+ license: apache-2.0
166
+ tags:
167
+ - reachy_mini_community_moves
168
+ ---
169
+
170
+ # Hand-written title
171
+
172
+ Precious hand-written body.
173
+ `;
174
+ HF.fetchRepoText = async (repoId, path) => (path === "README.md" ? custom : null);
175
+ const move = makeMove({ sharedStem: "dance-2" }); // no rename
176
+ await HF.uploadMove(move, "My Folder", "tok");
177
+
178
+ const up = calls.find((c) => c.fn === "uploadFiles");
179
+ const readme = await fileText(up, "README.md");
180
+ expect(readme).toContain("Precious hand-written body.");
181
+ expect(readme).toContain("# Hand-written title");
182
+ expect(readme).toContain("configs:");
183
+ });
184
+
185
+ it("README already viewer-ready and unchanged → not re-uploaded", async () => {
186
+ const move = makeMove({ sharedStem: "dance-2" });
187
+ // First share produces the ensured README; feed it back as the
188
+ // upstream one — the second share must skip the README write.
189
+ await HF.uploadMove(makeMove({ sharedStem: "dance-2" }), "My Folder", "tok");
190
+ const first = calls.find((c) => c.fn === "uploadFiles");
191
+ const ensured = await fileText(first, "README.md");
192
+
193
+ resetHubStub();
194
+ HF.fetchRepoText = async (repoId, path) => (path === "README.md" ? ensured : null);
195
+ await HF.uploadMove(move, "My Folder", "tok");
196
+ const up = calls.find((c) => c.fn === "uploadFiles");
197
+ expect(up.arg.files.map((f) => f.path)).not.toContain("README.md");
198
+ });
199
+
200
+ it("silent-only repo (no audio anywhere): no configs, no metadata.jsonl", async () => {
201
+ HF.listDatasetFiles = async () => [{ path: "data/other.json", _prefix: "data" }];
202
+ const move = makeMove({ audioBlob: null, audioMime: null, sharedStem: "dance-2" });
203
+ await HF.uploadMove(move, "My Folder", "tok");
204
+
205
+ const up = calls.find((c) => c.fn === "uploadFiles");
206
+ const paths = up.arg.files.map((f) => f.path);
207
+ expect(paths).not.toContain("data/metadata.jsonl");
208
+ const readme = await fileText(up, "README.md");
209
+ expect(readme).toContain("reachy_mini_community_moves");
210
+ expect(readme).not.toContain("configs:");
211
+ });
212
+
213
+ it("repo-state fetch failure: the share still goes through, README untouched", async () => {
214
+ HF.fetchRepoText = async () => { throw new Error("HF 500"); };
215
+ const move = makeMove({ sharedStem: "dance-2" });
216
+ const url = await HF.uploadMove(move, "My Folder", "tok");
217
+
218
+ expect(url).toContain("testuser/my-folder");
219
+ const up = calls.find((c) => c.fn === "uploadFiles");
220
+ const paths = up.arg.files.map((f) => f.path);
221
+ expect(paths).toContain("data/dance-2.json");
222
+ expect(paths).not.toContain("README.md");
223
+ expect(paths).not.toContain("data/metadata.jsonl");
224
+ });
225
+
226
+ it("sibling moves' descriptions land in metadata via opts.localMoves", async () => {
227
+ HF.listDatasetFiles = async () => [
228
+ { path: "data/other.json", _prefix: "data" },
229
+ { path: "data/other.ogg", _prefix: "data" },
230
+ ];
231
+ const move = makeMove({ sharedStem: "dance-2" });
232
+ await HF.uploadMove(move, "My Folder", "tok", {
233
+ localMoves: [{ label: "Other", sharedStem: "other", description: "The other move." }],
234
+ });
235
+
236
+ const up = calls.find((c) => c.fn === "uploadFiles");
237
+ const meta = await fileText(up, "data/metadata.jsonl");
238
+ expect(meta).toContain(`"file_name":"dance-2.ogg"`);
239
+ expect(meta).toContain(`"file_name":"other.ogg"`);
240
+ expect(meta).toContain(`"description":"The other move."`);
241
+ });
242
+ });
tests/unit/hf-viewer-sweep.test.js ADDED
@@ -0,0 +1,116 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ // repairOwnedViewerConfigs — the startup auto-repair sweep. It pushes
2
+ // commits unattended, so its guards get pinned here: own-namespace only,
3
+ // skip already-fixed repos, skip no-audio repos, byte-preserving README
4
+ // edit, per-repo error isolation.
5
+
6
+ import { describe, it, expect, beforeEach, afterEach } from "vitest";
7
+ import { HF } from "../../lib/hf.js";
8
+ import { calls, behavior, resetHubStub } from "../stubs/hf-hub.js";
9
+
10
+ const README = `---
11
+ license: apache-2.0
12
+ tags:
13
+ - reachy_mini_community_moves
14
+ ---
15
+
16
+ # Vintage dataset
17
+
18
+ Hand-written notes.
19
+ `;
20
+
21
+ const origNamespace = HF.getHfNamespace;
22
+ const origList = HF.listDatasetFiles;
23
+ const origFetchText = HF.fetchRepoText;
24
+
25
+ beforeEach(() => {
26
+ resetHubStub();
27
+ HF.getHfNamespace = async () => "testuser";
28
+ HF.listDatasetFiles = async () => [
29
+ { path: "data/wave.json", _prefix: "data" },
30
+ { path: "data/wave.ogg", _prefix: "data" },
31
+ ];
32
+ HF.fetchRepoText = async (repoId, path) => (path === "README.md" ? README : null);
33
+ });
34
+
35
+ afterEach(() => {
36
+ HF.getHfNamespace = origNamespace;
37
+ HF.listDatasetFiles = origList;
38
+ HF.fetchRepoText = origFetchText;
39
+ });
40
+
41
+ const repoMap = (entries) => new Map(entries);
42
+
43
+ describe("repairOwnedViewerConfigs", () => {
44
+ it("fixes an owned repo missing the config: one commit, README preserved + metadata", async () => {
45
+ const res = await HF.repairOwnedViewerConfigs({
46
+ token: "tok",
47
+ repos: repoMap([["testuser/my-moves", [{ label: "Wave", sharedStem: "wave", description: "Hi!" }]]]),
48
+ });
49
+ expect(res).toEqual({ checked: 1, fixed: 1, errors: 0 });
50
+
51
+ const up = calls.find((c) => c.fn === "uploadFiles");
52
+ expect(up.arg.commitTitle).toBe("Marionette: enable dataset-viewer audio previews");
53
+ const readme = await up.arg.files.find((f) => f.path === "README.md").content.text();
54
+ expect(readme).toContain("Hand-written notes.");
55
+ expect(readme).toContain("configs:");
56
+ expect(readme).toContain(`- "data/*.ogg"`);
57
+ const meta = await up.arg.files.find((f) => f.path === "data/metadata.jsonl").content.text();
58
+ expect(meta).toContain(`"file_name":"wave.ogg"`);
59
+ expect(meta).toContain(`"description":"Hi!"`);
60
+ });
61
+
62
+ it("skips repos outside the user's namespace entirely", async () => {
63
+ const res = await HF.repairOwnedViewerConfigs({
64
+ token: "tok",
65
+ repos: repoMap([["pollen-robotics/emotions", []]]),
66
+ });
67
+ expect(res).toEqual({ checked: 0, fixed: 0, errors: 0 });
68
+ expect(calls.filter((c) => c.fn === "uploadFiles")).toHaveLength(0);
69
+ });
70
+
71
+ it("skips repos whose README already has configs (idempotent)", async () => {
72
+ HF.fetchRepoText = async (repoId, path) =>
73
+ (path === "README.md" ? `---\nconfigs:\n - x\n---\nbody` : null);
74
+ const res = await HF.repairOwnedViewerConfigs({
75
+ token: "tok",
76
+ repos: repoMap([["testuser/my-moves", []]]),
77
+ });
78
+ expect(res).toEqual({ checked: 1, fixed: 0, errors: 0 });
79
+ expect(calls.filter((c) => c.fn === "uploadFiles")).toHaveLength(0);
80
+ });
81
+
82
+ it("skips repos with no audio (a config there would break the viewer)", async () => {
83
+ HF.listDatasetFiles = async () => [{ path: "data/silent.json", _prefix: "data" }];
84
+ const res = await HF.repairOwnedViewerConfigs({
85
+ token: "tok",
86
+ repos: repoMap([["testuser/my-moves", []]]),
87
+ });
88
+ expect(res).toEqual({ checked: 1, fixed: 0, errors: 0 });
89
+ expect(calls.filter((c) => c.fn === "uploadFiles")).toHaveLength(0);
90
+ });
91
+
92
+ it("isolates per-repo failures and keeps sweeping", async () => {
93
+ let call = 0;
94
+ behavior.uploadFiles = async () => { if (++call === 1) throw new Error("boom"); };
95
+ const res = await HF.repairOwnedViewerConfigs({
96
+ token: "tok",
97
+ repos: repoMap([
98
+ ["testuser/broken", []],
99
+ ["testuser/fine", []],
100
+ ]),
101
+ });
102
+ expect(res).toEqual({ checked: 2, fixed: 1, errors: 1 });
103
+ });
104
+
105
+ it("generates a community README when the repo has none", async () => {
106
+ HF.fetchRepoText = async () => null;
107
+ await HF.repairOwnedViewerConfigs({
108
+ token: "tok",
109
+ repos: repoMap([["testuser/my-moves", []]]),
110
+ });
111
+ const up = calls.find((c) => c.fn === "uploadFiles");
112
+ const readme = await up.arg.files.find((f) => f.path === "README.md").content.text();
113
+ expect(readme).toContain("reachy_mini_community_moves");
114
+ expect(readme).toContain("configs:");
115
+ });
116
+ });
tests/unit/viewer-config.test.js ADDED
@@ -0,0 +1,184 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ // Dataset-viewer support helpers (docs/hf-viewer-fix-spec.md):
2
+ // viewerConfigPlan (glob planning per layout/extension), ensureViewerConfig
3
+ // (byte-preserving frontmatter surgery), metadata.jsonl building/parsing.
4
+
5
+ import { describe, it, expect } from "vitest";
6
+ import {
7
+ viewerConfigPlan,
8
+ renderViewerConfigBlock,
9
+ ensureViewerConfig,
10
+ hasViewerConfig,
11
+ parseMetadataJsonl,
12
+ buildViewerMetadataRows,
13
+ buildMetadataJsonl,
14
+ } from "../../lib/hf.js";
15
+
16
+ describe("viewerConfigPlan", () => {
17
+ it("emits one glob per audio extension present, data/ layout", () => {
18
+ const plan = viewerConfigPlan([
19
+ "data/a.json", "data/a.ogg", "data/b.json", "data/b.wav", "README.md",
20
+ ]);
21
+ expect(plan.prefix).toBe("data/");
22
+ expect(plan.dataFiles).toEqual(["data/*.ogg", "data/*.wav", "data/metadata.jsonl"]);
23
+ expect(plan.metadataPath).toBe("data/metadata.jsonl");
24
+ });
25
+
26
+ it("supports root layout (Python-seeded libraries)", () => {
27
+ const plan = viewerConfigPlan(["a.json", "a.ogg", "README.md"]);
28
+ expect(plan.prefix).toBe("");
29
+ expect(plan.dataFiles).toEqual(["*.ogg", "metadata.jsonl"]);
30
+ });
31
+
32
+ it("returns null when the repo has no audio (rule 1: unmatched globs break the viewer)", () => {
33
+ expect(viewerConfigPlan(["data/a.json", "README.md", "marionette.json"])).toBeNull();
34
+ expect(viewerConfigPlan([])).toBeNull();
35
+ });
36
+
37
+ it("never treats .json/.jsonl or non-audio files as audio", () => {
38
+ const plan = viewerConfigPlan(["data/a.ogg", "data/metadata.jsonl", "data/b.json", "data/c.txt"]);
39
+ expect(plan.dataFiles).toEqual(["data/*.ogg", "data/metadata.jsonl"]);
40
+ });
41
+
42
+ it("prefers the data/ layout when both layouts hold audio", () => {
43
+ const plan = viewerConfigPlan(["rogue.wav", "data/a.ogg"]);
44
+ expect(plan.prefix).toBe("data/");
45
+ expect(plan.dataFiles).toEqual(["data/*.ogg", "data/metadata.jsonl"]);
46
+ });
47
+ });
48
+
49
+ describe("ensureViewerConfig", () => {
50
+ const FILES = ["data/*.ogg", "data/metadata.jsonl"];
51
+ const README = `---
52
+ license: apache-2.0
53
+ tags:
54
+ - reachy_mini_community_moves
55
+ pretty_name: My Moves • Reachy Mini Moves
56
+ ---
57
+
58
+ # My Moves
59
+
60
+ Hand-written body that must never change.
61
+ `;
62
+
63
+ it("inserts a configs block into existing frontmatter, preserving everything else", () => {
64
+ const out = ensureViewerConfig(README, FILES);
65
+ expect(hasViewerConfig(out)).toBe(true);
66
+ expect(out).toContain(`- "data/*.ogg"`);
67
+ expect(out).toContain(`- "data/metadata.jsonl"`);
68
+ // Byte-for-byte preservation outside the block: removing the block
69
+ // lines must give back the original.
70
+ const stripped = out.split("\n")
71
+ .filter((l, i, arr) => {
72
+ const block = renderViewerConfigBlock(FILES).split("\n");
73
+ const start = arr.indexOf("configs:");
74
+ return i < start || i >= start + block.length;
75
+ })
76
+ .join("\n");
77
+ expect(stripped).toBe(README);
78
+ });
79
+
80
+ it("is idempotent", () => {
81
+ const once = ensureViewerConfig(README, FILES);
82
+ expect(ensureViewerConfig(once, FILES)).toBe(once);
83
+ });
84
+
85
+ it("updates an existing configs block in place (new extensions appear)", () => {
86
+ const once = ensureViewerConfig(README, ["data/*.wav", "data/metadata.jsonl"]);
87
+ const twice = ensureViewerConfig(once, FILES);
88
+ expect(twice).not.toContain(`"data/*.wav"`);
89
+ expect(twice).toContain(`"data/*.ogg"`);
90
+ // Still exactly one configs key.
91
+ expect(twice.match(/^configs:/gm)).toHaveLength(1);
92
+ });
93
+
94
+ it("replaces a configs block that sits before other keys, keeping them", () => {
95
+ const readme = `---
96
+ configs:
97
+ - config_name: default
98
+ data_files:
99
+ - split: train
100
+ path:
101
+ - "data/*.wav"
102
+ tags:
103
+ - reachy_mini_community_moves
104
+ ---
105
+
106
+ body
107
+ `;
108
+ const out = ensureViewerConfig(readme, FILES);
109
+ expect(out).toContain("tags:\n - reachy_mini_community_moves");
110
+ expect(out).toContain(`"data/*.ogg"`);
111
+ expect(out).not.toContain(`"data/*.wav"`);
112
+ expect(out).toContain("\nbody\n");
113
+ });
114
+
115
+ it("returns the README untouched when there is no audio", () => {
116
+ expect(ensureViewerConfig(README, [])).toBe(README);
117
+ expect(ensureViewerConfig(README, null)).toBe(README);
118
+ });
119
+
120
+ it("prepends frontmatter when the README has none", () => {
121
+ const out = ensureViewerConfig("# Just a body\n", FILES);
122
+ expect(out.startsWith("---\nconfigs:")).toBe(true);
123
+ expect(out.endsWith("# Just a body\n")).toBe(true);
124
+ });
125
+ });
126
+
127
+ describe("hasViewerConfig", () => {
128
+ it("detects a top-level configs key only", () => {
129
+ expect(hasViewerConfig("---\nconfigs:\n - x\n---\nbody")).toBe(true);
130
+ expect(hasViewerConfig("---\ntags:\n - configs: nope\n---\n")).toBe(false);
131
+ expect(hasViewerConfig("no frontmatter, mentions configs:")).toBe(false);
132
+ expect(hasViewerConfig("")).toBe(false);
133
+ });
134
+ });
135
+
136
+ describe("metadata.jsonl", () => {
137
+ it("parses rows and skips malformed lines", () => {
138
+ const map = parseMetadataJsonl(
139
+ `{"file_name": "a.ogg", "title": "a"}\nnot json\n\n{"file_name": "b.ogg", "description": "d"}\n`,
140
+ );
141
+ expect([...map.keys()]).toEqual(["a.ogg", "b.ogg"]);
142
+ expect(map.get("b.ogg").description).toBe("d");
143
+ });
144
+
145
+ it("builds rows from local moves, existing rows, and stems — no `label` column ever", () => {
146
+ const paths = ["data/dance.json", "data/dance.ogg", "data/mystery.wav", "data/silent.json"];
147
+ const plan = viewerConfigPlan(paths);
148
+ const rows = buildViewerMetadataRows({
149
+ paths,
150
+ plan,
151
+ localMoves: [
152
+ { label: "Dance", sharedStem: "dance", description: "A great dance." },
153
+ ],
154
+ existingRows: new Map([["mystery.wav", { file_name: "mystery.wav", description: "kept from before" }]]),
155
+ });
156
+ expect(rows).toEqual([
157
+ { file_name: "dance.ogg", title: "dance", description: "A great dance.", motion_file: "dance.json" },
158
+ { file_name: "mystery.wav", title: "mystery", description: "kept from before" },
159
+ ]);
160
+ for (const r of rows) expect("label" in r).toBe(false);
161
+ });
162
+
163
+ it("maps imported moves by their upstream stem (id tail)", () => {
164
+ const paths = ["data/amazed1.json", "data/amazed1.ogg"];
165
+ const plan = viewerConfigPlan(paths);
166
+ const rows = buildViewerMetadataRows({
167
+ paths,
168
+ plan,
169
+ localMoves: [{ id: "someone/repo/amazed1", fromCommunity: true, label: "Amazed", description: "Wow." }],
170
+ });
171
+ expect(rows).toEqual([
172
+ { file_name: "amazed1.ogg", title: "amazed1", description: "Wow.", motion_file: "amazed1.json" },
173
+ ]);
174
+ });
175
+
176
+ it("serializes one JSON object per line with a trailing newline", () => {
177
+ const text = buildMetadataJsonl([
178
+ { file_name: "a.ogg", title: "a" },
179
+ { file_name: "b.ogg", title: "b" },
180
+ ]);
181
+ expect(text).toBe(`{"file_name":"a.ogg","title":"a"}\n{"file_name":"b.ogg","title":"b"}\n`);
182
+ expect(buildMetadataJsonl([])).toBe("");
183
+ });
184
+ });
views/folder.js CHANGED
@@ -686,6 +686,10 @@ export function createFolder({
686
  const url = await HF.uploadMove(m, repoName, robot._token, {
687
  audioOffsetMs: folder?.audioOffsetMs,
688
  protectedStems,
 
 
 
 
689
  });
690
  m.shared = true;
691
  m.sharedUrl = url;
 
686
  const url = await HF.uploadMove(m, repoName, robot._token, {
687
  audioOffsetMs: folder?.audioOffsetMs,
688
  protectedStems,
689
+ // Feed the folder's moves to the metadata.jsonl
690
+ // builder so every audio row in the repo keeps a
691
+ // title/description, not just the move being shared.
692
+ localMoves: siblings,
693
  });
694
  m.shared = true;
695
  m.sharedUrl = url;