tracker

7.3 KB

1. # w061 — tracker#28: full cast (all seasons) + crew
3. **Commit**: `10bb3f9` on branch `t26` (worktree loreana:/media/STORAGE/projects/tracker-wt26, on top of #26 `25a50bc`). Not pushed, not merged, not deployed. main untouched.
5. ## Measured first (live copy)
6. - The page showed `show.cast`. That is the old tracker's migrated list for 1,820 of 10,011 titles; the rest had nothing. #26 added TMDB's first 20 only for filmography titles. No crew anywhere.
7. - **Star Trek: Picard: 4** (Stewart, Ryan, Hurd, Speleers). **Superman (1978): 0.** Dune (1984): 39.
9. ## Built
10. - **Stored** (`details.hl`): from a details answer with `credits` (movie) / `aggregate_credits` (series = every season):
11. - **Cast**: TMDB's order (a series' is by episode count); roles split, each once, at most 4 + "…"; at most 300 per title. `guest` = fewer than half the series' episodes (at most 10). Adult people are left out.
12. - **Crew**: created_by, Director, Writer, Screenplay, Story, Composer/Music. Grouped by job, one entry per person, at most 30 per job.
13. - Every person is linked. Unknown people get a minimal person record (the #26 `castPersonId`). `creditsAt` marks a title as done.
14. - **Who fills it**:
15. - **#26 completion**: the same request as before.
16. - **Daily sync and `tools/sync-tmdb.hl`**: the details request now appends the credits (no extra request).
17. - **Search import**: one details request, handed to `syncShowWith` (one request fewer than before); the face stores the credits. This closes w059's open point.
18. - **Credits job** (`project.hl creditsTick`): runs after the backfill and the repair, only public titles, one request + 260 ms per title. It resumes after a restart, marks 404/adult titles, and retries passing failures. It persists only shows + people. It **pauses above `TRACKER_CREDITS_MAX_MB` (16000)**.
19. - **Show page** (mobile first):
20. - Crew block with job labels, then "Cast": `Name (Character)`, a "Guest stars" label before the guests.
21. - Cast collapsed after 20 with "Show all (N)". Crew shows 6 per job, then "Show all crew (N)".
22. - Both toggles run client-side; no face is sent.
23. - **`shows.hl showBySlug`**: now a slug → id map. The old version scanned and decoded every title on each call, twice per show page. That cost 82 ms per call before any credits, grew with them, and leaked memory.
25. ## Gate
26. ```
27. TRACKER_GATE_SHOTS=/tmp/w061/shots TRACKER_GATE_PORT=8754 TRACKER_GATE_IDENT_PORT=8755 TRACKER_GATE_CHROME=8756-8758 TRACKER_GATE_TMDB_PORT=8759 node tests/browser.mjs
28. 249 passed, 0 failed
29. ```
30. 231 → 249 checks. The new ones cover:
31. - series aggregate cast incl. a later-season actor, roles cleaned up, collapsed to 20, "Show all" without a request, "Guest stars";
32. - crew order and dedup;
33. - movie director/writer/screenplay/story/music and the crew toggle;
34. - the search-imported title has cast + crew;
35. - the #26-completed title shows its crew;
36. - the sync tool stores credits and its second run adds nobody;
37. - the credits job: pause at the memory limit, stop/resume, done, latency < 1.2 s, nothing left on the next start.
39. ## Real data (copy, real TMDB; deleted afterwards)
40. - **Picard: 4 → 202 cast** (17 main incl. the season-3 regulars, then guests) **+ 45 crew**.
41. - **Superman: 0 → 90 cast**, Directed by Donner, Written by Mankiewicz, Screenplay/Story Puzo et al., Music John Williams.
42. - **Credits job: 5,867 titles in ~51 min**, 0 failed:
43. - run 1 did 714 titles and was stopped;
44. - run 2 resumed: `done: 5153/5153 … cast=262634 crew=39624 people=104614 … seconds=2681`;
45. - the next start: "nothing to do".
46. - The details repair before it: 557 titles in 800 s.
47. - **Pages during run 2** (n=4,836): median 28 ms, p95 194 ms, p99 436 ms, max 1.1 s.
48. - **After**:
49. - show page full load: 105–235 ms (Picard HTML 166 kB);
50. - "Show all": 2–18 ms;
51. - signed-in watch click: Picard 184–200 ms, The Simpsons 276–731 ms.
52. - **Screenshots** (looked at):
53. - `/tmp/w061/real/{Star-Trek-Picard,Superman,Dune,The-Simpsons}-{collapsed,all}-{390,1280}.png`;
54. - gate: `/tmp/w061/shots/credits-*.png`.
56. ## Merge notes for main
57. - **`components/show.hl`:**
58. - statics after `genreRowsOf` (replacing `castRowsOf`);
59. - `completedPageOf`;
60. - members after `hasCast`;
61. - the View's crew + cast blocks (replacing the old cast loop);
62. - `completeTick`;
63. - 2 handlers before `toggleEpisode`.
64. - **`styles.hl`:** the cast/crew block replaces the two `.artist-link` lines.
65. - **`shows.hl`:** `showBySlug`.
66. - **`tmdbsync.hl`:** details paths + `res.details`.
67. - **`search.hl`:** `importTitle`.
68. - **`project.hl`:** the `readFile` import, `applyCredits` in the sync tick, the credits block, the clock.
69. - **Gate:** the sync, import and failed-import expectations now include `aggregate_credits`/`credits`.
70. - **Deploy:** the repair takes ~13 min, then the credits job ~51 min. Watch `docker stats`.
72. ## Open points
73. 1. Merge + deploy.
74. 2. **Hybriel memory leak (new, not filed):** the native runtime never frees fetched bodies or decoded records.
75. - Repro: 300 × `fetch` of a 440 KB JSON + `.json()` → RSS +2.7 GB; without `.json()` +0.43 GB.
76. - Run 1 reached 20 GB in 20 min, mostly from the old per-call `showBySlug` scans.
77. - The live container already used 11.4 GiB today.
78. 3. **Cost of 160k people** (was 15k): the search index at boot takes 4.3 s (was 0.66 s), and RSS after boot is 4.9 GB (was 1.6 GB). The alternative would be to not create people for guests (not built).
79. 4. Session faces (watch/follow) still re-derive the cast rows, since they come from `showRow`.
80. 5. Only followed titles refresh their credits (daily sync).
82. ## Hybriel issues
83. - The memory leak (open point 2).
84. - Note: strings have no `\n` escape, so `indexOf('\n')` looks for a literal backslash-n. Already documented in the guide (hybriel#35).
86. ## Places that create show / person records on t26 (for wiring `claimShortId` from main's #27 in the merge)
87. Every runtime path goes through one of three `put()` calls, in the record literal just before the call:
88. - **`people.hl` `castPersonId` (~line 349–355, `personsTable.put(record)`)** → `claimShortId('person', …)`. This is the only place that creates PEOPLE at runtime. It is used by:
89. - #26 `details.hl` `mergedCast` (completion of filmography titles);
90. - #28 `details.hl` `linkedCrew` (crew) and `withCredits` → completion, the credits job (`creditsStep`), the daily sync and `tools/sync-tmdb.hl` (`applyCredits`), the search import face (`components/search.hl searchImport` → `applyCredits`).
91. - The live-copy credits run created ~150k people through it.
92. - It also writes `personTmdbIds`, `slugIndex.ids` and `indexPerson` after the put.
93. - **`people.hl` `addMinimalTitle` (~line 248–256, `showsTable.put(record)`)** → `claimShortId('show', …)`. Filmography titles (#16; `minimal = true`, completed by #26).
94. - **`search.hl` `importTitle` (~line 390–400, `showsTable.put(record)`)** → `claimShortId('show', …)`. Search import. On t26 the request path changed (one details request with `external_ids,aggregate_credits|credits`, then `syncShowWith`); the record literal is otherwise main's.
95. - `tools/migrate.hl` (persons ~204, shows ~242): unchanged from main, one-off.
96. - Tests: `tests/seed-show.hl` / `tests/seed-adult.hl` write fixture records directly (unchanged from main); if main's gate expects shortIds there, the merge keeps main's versions.
98. No other code creates show/person records. Seasons and episodes are created in `tmdbsync.hl` (`syncShowPart`, `mergeTvmaze`) — not people or shows.