tracker

10.1 KB

1. # Report 070 — tracker: t18 + t20 + t19 merged into main, migrated data kept, Hybriel 8efba065
3. Repo: loreana:/media/STORAGE/projects/tracker.worldapi.org. Everything is committed on main (HEAD `8081350`). Not pushed, not deployed. The worktrees tracker-wt18, -wt20 and -wt19 and the branches t18, t20 and t19 are removed; they held nothing uncommitted (only `bin/` and `.scratch/`).
5. ## Commits (main)
6. | Commit | What |
7. |---|---|
8. | `20e09d8` | merge t18 (#18) |
9. | `daf49fe` | merge t20 (#20/#21) |
10. | `749019b` | merge t19 (#19) |
11. | `46b2138` | migrated data is never destroyed |
12. | `1ad19c8` | Hybriel re-vendor, `&` on read-only lambda parameters, merge sort on /my/unwatched, deploy.sh runs all 3 gates |
13. | `8081350` | docs (README, STATUS, docs/kinds.md, docs/franchises.md), real-data scripts, tools |
15. ## Conflicts and how they were resolved (both sides kept everywhere)
16. - **t18** conflicted in README, STATUS, show.hl, project.hl and browser.mjs.
17. - `pageShowOf` (the small object from mission 018 (old 062)) now uses `summaryOf`.
18. - Imports and boot steps are joined.
19. - **t20** conflicted in README, STATUS, show.hl, project.hl, search.hl, components/search.hl, browser.mjs and faketmdb.mjs.
20. - `slugInfoOf` sits beside 062's slug helpers.
21. - The shared clock runs: adult backfill → repair → **kinds** → credits.
22. - `withDetailsFields` also stores `tmdbType` and `kind`, so the delta light step and the #26 completion set them too.
23. - The search import keeps both `path` and `details`.
24. - browser.mjs: URLs changed to `/series/` and `/movies/`, h1 selectors to the typed `type-name`.
25. - kinds.mjs: repair and credits switched off, and its fake TMDB name set to the seed's long name. Otherwise the #18 rename hit it.
26. - **t19** conflicted in README, STATUS, show.hl, project.hl, browser.mjs and faketmdb.mjs.
27. - The franchise/timeline pages get the typed headings "Franchises", "Franchise | …", "Timeline | …" (also the timeline editor).
28. - Their title links go through `titlePath`.
29. - The collection seed now waits for every other job, so it runs last.
30. - franchises.mjs: URLs updated and 2 new checks (headings with their colours, mixed-timeline links).
31. - deploy.sh now runs browser.mjs, kinds.mjs and franchises.mjs. The default ports of kinds and franchises moved to 8700–8710, because kinds' old defaults 8765/8766 are taken by other services on Loreana.
33. ## Step 2: migrated data is kept (conductor's change)
34. - The one-time summary step now **moves** a copied summary to `migratedSummary` and empties `summary`. The marker is `summaries-moved.txt`.
35. - On a migrated record (oldId not `tmdb-…`), the first time TMDB replaces a value the old one is kept, and only that first time:
36. - shows: `migratedTitle`, `migratedGenres`, `migratedHomepage`, `migratedTagline`;
37. - seasons and episodes (oldId not `tmdb-`/`tvmaze-`): `migratedTitle`, `migratedSummary`.
38. - Not kept: status, counts, ids and the poster. They are TMDB's state and the sync has written them since #9.
39. - **Gate**: 3 checks.
40. - The first start moves the copied summary.
41. - The renamed show, the renamed movie and episode 1x1 keep their old values.
42. - A second rename keeps the first value.
43. - **Real copy** (`tools/count-migrated.hl`):
45. | Kept | Count |
46. |---|---|
47. | summaries | 9,647 |
48. | titles | 16 (e.g. "The Daily Show with Trevor Noah → The Daily Show") |
49. | genres | 71 |
50. | taglines | 16 |
51. | homepages | 4 |
52. | episode titles / summaries | 635 / 409 |
53. | season titles / summaries | 3 / 3 |
55. ## Hybriel
56. - **Commit**: master `8efba065` (#126 f685f240, #48 4371b7aa).
57. - `bin/hybriel` sha256 `50361e954e317a819aa58adfc2e3553f0b4b7adc82abd1fad79385d33b3d5b72`.
58. - Plugins updated: core crypto data fetch fs http http1 mpackdb proc smtp time web.
59. - The previous build is in `.scratch/pre-070/`.
60. - **w068's `/shows/<slug>` route** still works: `page(m, s, host, hdrs)`, `hostOf`, `requestHeaders` and the sessions API are unchanged. The kinds gate (function-rendered show page: season toggle, Follow) passes.
61. - **Lambda audit (#48)**, with `tools/lambda-audit.py`: 13 lambdas change a parameter without `&`.
62. - details.hl: `withCredits` :236, `withDetailsFields` :265, `addSum` :359, `firstStep` :412.
63. - franchises.hl: `storeOrder` :373.
64. - people.hl: `startFill` :222, `fillStep` :269.
65. - tmdbsync.hl: `mergeTvmaze` :279, `syncMovie` :442, `addTotals` :826.
66. - Every caller uses the returned value or the stored record, so no fix was needed.
67. - **Speed**: the copy per call was the real cost. A micro test passing a 20k-row list 2,000×: 15.5 s with the copy, 0.015 s with `&`.
68. - All 319 read-only parameters got `&` (`tools/ref-params.py`).
69. - /my/schedule went from 2.35 to 0.46 s, /my/unwatched from 13.7 to 5.7 s.
70. - /my/unwatched also got one merge sort instead of n² inserts: 5.7 → 4.8 s.
72. ## Gate commands (final tree, new binary)
73. ```
74. TRACKER_GATE_SHOTS=/tmp/w070/new-shots TRACKER_GATE_PORT=8751 TRACKER_GATE_IDENT_PORT=8752 TRACKER_GATE_CHROME=8753-8756 TRACKER_GATE_TMDB_PORT=8757 node tests/browser.mjs
75. 325 passed, 0 failed
76. TRACKER_GATE_SHOTS=/tmp/w070/new-kinds TRACKER_GATE_PORT=8750 TRACKER_GATE_IDENT_PORT=8751 TRACKER_GATE_CHROME=8752-8756 TRACKER_GATE_TMDB_PORT=8757 node tests/kinds.mjs
77. 32 passed, 0 failed
78. TRACKER_GATE_SHOTS=/tmp/w070/new-fr TRACKER_GATE_PORT=8750 TRACKER_GATE_IDENT_PORT=8751 TRACKER_GATE_CHROME=8752-8756 TRACKER_GATE_TMDB_PORT=8757 node tests/franchises.mjs
79. 50 passed, 0 failed
80. ```
82. ## Real data (copy of live storage/ 2026-10-02, real TMDB/TVmaze, all jobs, the daily run forced as a delta)
83. **Jobs on the first start, about 50 min in all:**
84. - The summary move: 16 s at boot.
85. - Kind backfill: 1,789 titles in 16–21 min.
86. - Collection seed: 4,774 movies in 32–37 min, giving 110 timelines.
87. - Adult backfill, repair and credits: nothing to do.
88. - One delta run: 261 titles in 6–10 min.
89. - Without `sync-state.txt`, the first 04:00 run after the deploy is a full walk.
91. **RSS old vs new** (controlled, one process at a time; "after jobs" = 200 page loads on a fresh process):
93. | | after boot | jobs peak | 200 loads after the jobs | /my/unwatched |
94. |---|---|---|---|---|
95. | old 8590df63 | 8.0 GB | 15.2 GB | 8.0 → 12.2 GB | 1.15 s |
96. | new, final tree | 2.1 GB | 11.8 GB (median 2.7) | 2.1 → 2.8 GB | 4.8 s |
97. | new, `HL_GC_BYTES=256 MiB` | 2.7 GB | 12.4 GB at 24 min (run stopped) | 2.7 → 3.5 GB | 2.2 s |
98. | new, `HL_GC_BYTES=0` | 6.2 GB | — | 6.2 → 8.7 GB | 1.15 s |
100. **Load times** (final tree, fresh process):
101. - curl: `/` 0.02 s, lists 0.01–0.05 s, `/franchises` 0.04 s, a series 0.25–0.45 s, /my/series 0.4 s, /my/schedule 0.42 s, /my/unwatched 4.8 s.
102. - Browser at 390/1280 (signed in as az5b2):
104. | Page | Full load |
105. |---|---|
106. | lists | 39–146 ms |
107. | timeline | ~40 ms |
108. | First Contact with its widget | 136–346 ms |
109. | Picard (202 cast) | 192–368 ms |
110. | /my/series | 0.8–1.1 s |
111. | /my/unwatched | 5.6–6.1 s |
112. | the first `/` after boot | 15 s |
114. Nothing scrolls sideways and there were 0 console problems.
115. - I looked at the screenshots on Loreana: `/tmp/w070/real/*-{390,1280}.png` and the gate shots `/tmp/w070/new-*`.
117. ## Open points
118. 1. **Memory after the jobs (Hybriel, for the architect).** The process that ran the jobs keeps about 25 MB per page load: 3.0 → 8.1 → 13.6 → 18.9 GB over 600 loads, with /my/unwatched at 8.3 s. A fresh process keeps 3.5 MB per load; after one delta run, about 8 MB. The old binary kept about 21 MB per load from the start.
119. - Recommendation: restart the container once after `collections done`. A daily restart after the 04:00 run is probably needed (not built). This is in README "Deploy".
120. 2. **GC by bytes is slow with a big heap (Hybriel).** With a 2+ GB live heap, every 64 MiB of plugin data triggers a full collection: /my/unwatched went from 1.2 to 4.8 s.
121. - `HL_GC_BYTES` is not set. I tried 256 MiB (2.2 s), but that run reached 12.4 GB by 24 min and I stopped it. Its full peak was not measured. This is the creator's/architect's call.
122. - Repro: `tests/realdata-070-bench.sh`.
123. 3. **systemd-oomd killed my first parallel run** (old + new at once) at 05:38. Loreana was above 90 % memory + swap, with other workers' tests running too. All numbers above come from later runs, one process at a time.
124. 4. **Port clash.** At the start, a tickets gate held 8720–8725. The lead moved me to 8750–8759.
125. 5. **No real "series page with a widget".** The seed only makes movie timelines. The gate covers a series with a widget; on real data I checked a movie with the widget and a series with its cast.
126. 6. Still open from earlier missions: TMDB type Video → Series? (#21), #18's 24 kept "own" summaries, #19's rename/delete not built.
128. ## Where I stopped (wrap-up, usage limit)
129. The lead's messages about hybriel#127, the move to master d98926c6 and the port switch reached me only after this work was finished and committed. The tracker therefore runs on **8efba065**, not d98926c6.
131. **Still to do after the reset:**
132. 1. Re-vendor to d98926c6 or a later HEAD: same procedure, `.scratch/pre-070` stays as the rollback copy.
133. 2. Re-run the 3 gates on 8750–8757.
134. 3. Run the big-page measurement, old vs new, 20 loads each, time and RSS:
135. - /my/unwatched (~1,300 rows)
136. - Picard with "Show all"
137. - /series
138. - a franchise index or timeline
139. 4. Re-run the RSS and job measurements: `tests/realdata-070-run.sh` and `tests/realdata-070-bench.sh` (STATUS "How to repeat"), real-data server on 8758.
140. 5. Check whether #127 explains the per-load growth and the slow /my/unwatched (open points 1 and 2).
142. **Big-page numbers I already have.** Fresh process, curl, 10 loads per page, average:
144. | Page | old 8590df63 | new 8efba065 (final tree, with `&`) |
145. |---|---|---|
146. | /my/unwatched (1,287 rows) | 1.15 s | 4.8 s |
147. | /series/Star-Trek-Picard (collapsed cast) | 0.055 s | ~0.25 s |
148. | /series | 0.014 s | 0.018 s |
149. | /my/series (126 rows) | 0.20 s | 0.40 s |
151. - The process that had run the jobs got slower on every load, the #127 symptom: /my/unwatched 8.3 s, and RSS grew about 25 MB per load.
152. - Not yet measured: Picard with "Show all" and a franchise index.
154. ## Leftovers
155. - Nothing of mine is running. Ports 8720–8731 and 8750–8759 are free.
156. - The data copies, the old-binary tree and its `.env` link are deleted. I never read `.env`.
157. - Logs are in `.scratch/w070/`.