tracker

4.9 KB

1. # w054: tracker, adult titles hidden + adult-flag backfill (mission 010 (old 054))
3. The work is in loreana:/media/STORAGE/projects/tracker.worldapi.org, commit **`c171227`** on main. Nothing was pushed and nothing was deployed.
5. **What was built**
6. - Every title has `shows.adult`: `true`, `false`, or `null` for unknown.
7. - The homepage rows, `/movies`, `/shows` and the search (our database and "Fetch from web") show only titles where `adult == false`.
8. - Unknown titles are hidden too.
9. - The web search sends `include_adult=false` and also checks each result's own `adult` field.
10. - `/my/*` is not filtered, so an adult title the user follows stays there.
11. - The daily sync and the search import also store TMDB's adult flag.
12. - **The backfill** runs inside the app (`project.hl backfillTick` calls `tmdbsync.hl backfillTitle`):
13. - It handles one title per step, on the same clock as the daily sync. While the daily sync runs, the backfill waits.
14. - Per title it makes 1 TMDB details request (with external_ids) and downloads the poster if the file is missing. After each API request it pauses 260 ms.
15. - A 404 marks the title as `adultCheck` and it is not asked again. Passing failures are retried.
16. - After a restart it continues with the titles that are still unknown.
17. - It writes a log line every 500 titles and a done line at the end.
18. - `TRACKER_BACKFILL=0` switches it off.
20. **Gate**
21. `TRACKER_GATE_SHOTS=/tmp/w054/shots TRACKER_GATE_PORT=8720 TRACKER_GATE_IDENT_PORT=8721 TRACKER_GATE_CHROME=8722-8726 TRACKER_GATE_TMDB_PORT=8727 node tests/browser.mjs` → last line **`184 passed, 0 failed`** (167 existing checks + 17 new).
22. - Proven by the new checks:
23. - Titles with an unknown flag and adult titles are hidden.
24. - Non-adult titles are shown.
25. - A followed adult title is still on `/my/movies` and its page opens.
26. - The web search hides TMDB's adult result.
27. - The resume works: stopped after 6 titles, the restart asks only the other 8.
28. - Pages stayed under 1.2 s during the backfill (slowest 634 ms).
29. - The lists and the search update without a restart.
30. - A third start has nothing to do.
31. - Screenshot `/tmp/w054/shots/adult-home-1280.png` (I looked at it).
33. **Real data** (copy of the live `storage/`, real TMDB; the copy was deleted afterwards)
35. | | |
36. |---|---|
37. | Run time | 9,453 titles in **98.9 min** (5935 s) |
38. | Adult | **3,450** (3,448 movies, 2 series) |
39. | Not adult | **5,866** |
40. | Unknown left | **137**, all TMDB 404, stay hidden |
41. | Posters | **8,740** downloaded (126 → 8,866 files, 459 MB) |
42. | Errors | **140**: 137 × 404, 2 poster 404s, 1 TLS failure that was retried and then worked |
44. - A restart afterwards logs "nothing to do".
45. - After the run, `/movies` shows 4,242 movies and `/shows` 1,624 series.
46. - I checked every link on `/`, on all 245 list pages and in 10 searches (6,008 links, including "sex", "porn", "xxx"). All of them are titles marked `adult == false`.
47. - Page load times during the run, signed out, measured with curl about every 5 s:
49. | page | average | max |
50. |---|---|---|
51. | `/` | 0.156 s | 4.87 s |
52. | `/movies` | 0.059 s | 1.67 s |
53. | `/shows` | 0.045 s | 1.54 s |
54. | `/search/the` | 0.075 s | 1.33 s |
56. - 14 of 4,620 requests took more than 1 s. Each one happened during a slow TMDB or poster answer (13 steps over 1.5 s, the slowest 5.8 s).
57. - Screenshots on Loreana: `/tmp/w054/real/{home,movies,shows,search-the}{,-bottom}-{390,1280}.png`. I looked at them and saw no adult title or poster. Logs are in `/tmp/w054/real-*.log`.
59. **Open points**
60. 1. Deploy: the live app starts the backfill immediately. It takes about 100 min and adds about 450 MB of posters to `storage/`, which also goes into the nightly backup. At the first start the public lists are empty, and they fill up while the backfill runs.
61. 2. A slow TMDB answer blocks all pages while it lasts, because hl:fetch is synchronous. The worst seen was 4.9 s. Fixing it needs a non-blocking fetch in Hybriel or a shorter timeout (it is 20 s now).
62. 3. Posters of adult titles are downloaded too. Skipping them would save about 170 MB.
63. 4. **Hybriel issue:** I could not hide the show page of an adult title from people who don't follow it.
64. - Attempt: `found = showRow != null && !adultHiddenFor(showRow, me)`, where `adultHiddenFor` is a static that reaches mpackdb.
65. - Result: after any session face on the page (for example `showToggleEpisode`), the browser re-derives `found` and then `rows`, and `rows` fails with "'watchesOfUser' … realm 'client' cannot bind". Copying the value into a plain member (`hideAdult`) first did not help.
66. - So I reverted it. The page of an adult title opens for anyone who has the link, but it appears in no list and no search.
67. 5. 137 titles that TMDB no longer has stay hidden.
68. 6. Not mine, seen in passing: signed out at 390 px, the header buttons wrap to two lines. I also tarred the live `storage/` twice (read-only both times): the first `ssh` inside `ssh loreana bash -s` swallowed the rest of the script, so I re-ran it with `< /dev/null`. This is noted in STATUS.