tracker

5.7 KB

1. # w060: tracker#27, short ids (mission 016 (old 060))
3. **Commit**: `d8b12d6` on main in loreana:/media/STORAGE/projects/tracker.worldapi.org. Not pushed and not deployed. I did not touch tracker-wt26.
5. ## What was built
6. - Every show (movie or series) and every person has a field `shortId`: 5 characters from `[a-z0-9]`.
7. - Each short id is unique across both tables. One in-memory map in `shortids.hl` covers both tables and is built at start (0.12–0.22 s on real data).
8. - I did not use an mpackdb index, because an mpackdb index covers only one table.
9. - **Old ids**: `data/old-short-ids.json` maps the old tracker's 702 ids from oldId to id.
10. - `tools/old-short-ids.hl` made the file once from `Show.jsonl`. All 702 old ids are already `[a-z0-9]{5}`.
11. - The file is part of the code, so the deploy sends it along.
12. - All 702 old ids are reserved: a new random id is never one of them.
13. - **New records** get a short id when they are created (`claimShortId`): the search import and the titles the person fill adds.
14. - Those are the only places where the app creates shows.
15. - Persons are only created by `tools/migrate.hl`, and the sync only creates seasons and episodes.
16. - **Backfill**: chosen to run in the background, not at start.
17. - Doing all records at start would block the server for at least 16 s (25k records).
18. - It runs on its own 0.1 s clock and works in batches of up to 40 ms.
19. - It saves to disk every 20 batches and at the end.
20. - After a restart it continues with the records that still have no short id.
21. - If a stored short id is already held by another record, that record gets a new one (tested in the gate).
22. - `TRACKER_SHORTIDS=0` switches it off. `TRACKER_SHORTID_BUDGET_MS` and `TRACKER_SHORTID_LOG_EVERY` are its other settings.
23. - **Shown** as `#shortid` under the poster, and under the person's photo. A person without a photo has it under the name and facts. It is small (.8rem), muted and monospace.
24. - **Redirect**: `/<shortId>` answers 301 to `/shows/<slug>` or `/person/<slug>`. Upper case works too. Anything else answers 404. This is the last route, so every named route wins over it.
26. ## Gate
27. `TRACKER_GATE_SHOTS=/tmp/w060/shots TRACKER_GATE_PORT=8720 TRACKER_GATE_IDENT_PORT=8721 TRACKER_GATE_CHROME=8722-8726 TRACKER_GATE_TMDB_PORT=8727 node tests/browser.mjs`
28. → **`259 passed, 0 failed`** (was 238; 21 new checks). The gate takes about 1.5 min.
30. The new checks prove:
31. - The old id is kept: sh1 → 100jh.
32. - A movie and a series get new ids.
33. - A search import has its short id right away.
34. - The person fill's titles get theirs when they are created.
35. - The id is under the poster and the photo, and under the facts for a person without a photo. Checked: monospace, muted and smaller, at 390 and 1280 px.
36. - The 301 redirects and the 404s work, and the browser lands on the right page.
37. - `tools/count-short-ids.hl`: every record has a short id and there are 0 duplicates.
38. - Resume: `tests/seed-shortids.hl` adds 60 persons without an id and 2 that share `aaaa1`. A start with a 1 ms budget is stopped part way, and the next start does only the rest. One of the two keeps `aaaa1` and the other gets a new id. A third start has nothing to do.
39. - I looked at the screenshots in Loreana `/tmp/w060/shots/shortid-*`.
41. ## Real data
42. Copy of Byrodin `storage/` (10,010 shows, 15,157 persons), server on :8730.
44. | | |
45. |---|---|
46. | **Backfill duration** | 25,167 records in **297 s** (another run: 307 s) |
47. | Batches | 1,859; slowest 62 ms |
48. | Slowest save to disk | 31 ms |
49. | **Old kept** | **702 / 702** (Raised by Wolves `100jh`, Foundation `100sp`) |
50. | Lost | 0 |
51. | New ids that are an old one | 0 |
52. | **Duplicates across shows + persons** | **0** |
53. | Duplicate @ids (all tables, `count-duplicate-ids`) | 0 |
55. **Pages during the backfill** (curl, back to back, about 520 requests each; avg / max / number over 1 s):
57. | page | during the backfill | without it (baseline, 120 s) |
58. |---|---|---|
59. | `/shows/Foundation` | 0.234 / 1.56 / 9 | 0.173 / 1.34 / 5 |
60. | `/person/Jared-Harris` | 0.170 / 1.37 / 9 | 0.046 / 1.10 / 5 |
61. | `/` | 0.070 / 1.34 / 2 | 0.022 / 0.99 / 0 |
62. | `/movies` | 0.073 / 1.18 / 2 | 0.015 / 1.01 / 1 |
64. - Answers over 1 s happen without the backfill too. Loreana was busy with other workers.
65. - The backfill's own steps never took more than 62 ms. On average it adds about 0.1 s per page.
67. **Full page loads** (signed in as the creator, ms, 3 loads each):
69. | page | 390 px | 1280 px |
70. |---|---|---|
71. | Raised by Wolves (opened via `/100jh`) | 1372 / 297 / 310 | 335 / 266 / 301 |
72. | Foundation | 302 / 306 / 250 | 1246 / 205 / 195 |
73. | Star Trek: First Contact (`aafif`) | 275 / 287 / 275 | 211 / 235 / 212 |
74. | Jared Harris (`yg7lk`) | 164 / 138 / 122 | 104 / 119 / 121 |
76. - Every short id answers 301 in 1–27 ms, and the browser lands on the page. `/zzzzz` answers 404. There were 0 console problems.
77. - Screenshots on Loreana: `/tmp/w060/real/{old-id,series,movie,person}-{390,1280}.png`. I looked at them. That folder also has older `Grant-Gustin*`/`Jared-Harris*` files that are not mine.
78. - The copy and its sessions are deleted. Nothing of mine is still running on 8720–8731.
80. ## Open points
81. 1. Deploy: the first start runs the backfill (about 5 minutes). Short ids appear page by page while it runs.
82. 2. Inside the app, one record costs about 3 ms; in a plain script it costs about 0.6 ms. I did not find out why.
83. - **Hybriel finding**: reading a list kept as a field of a `static` object (`state.queue[i]`) seems to copy the whole list on every access. Batches got about 5× faster after I made the list a plain `static`. I made no minimal repro.
84. 3. Nothing in the app creates persons today. A later step that does must call `claimShortId('person', id, oldId)`.
85. 4. `/<shortId>` of an adult title still redirects. That page then says "Not found" to anyone who does not follow it, as before.