gitoriaLog in with ident

tracker

All repositories: gitoria

ReadmeCodePull requestsReleasesTicketsSettings
Commit7d7d44877d7d4487tracker#33 (mission 029): reduced titles — every title TMDB's details never went through this app (no detailsAt, no tmdbSync) is incomplete (shows.hl isIncomplete; the old tracker's migrated rows passed #26's test: 5,697 non-adult on the live copy, 691 series without seasons); the repair job does the visibly reduced first (shows.hl missingParts), the page completes one on open; a title TMDB has no poster for (The Remaining) shows the placeholder; tools/count-incomplete.hl; gate fixtures stand for synced titles (tmdbSync), tests/seed-reduced.hl + #33 checks; gates 347/0, 32/0, 52/0mre7d7d4487/lib/sync.hl

24.0 KB

  1. // lib/sync.hl — THE TMDB SYNC OF ONE TITLE (tracker.worldapi.org#9; tmdbsync.hl before mission 028 — its requests are in tmdb.hl,
  2. // its small helpers in sync-helpers.hl, the backfills in backfill.hl): for every show SOMEONE follows, fetch the show's
  3. // details, seasons and episodes from TMDB and bring storage/mpackdb/ up to date, plus the show's poster.
  4. // Used by BOTH runners: the daily run inside the app (jobs.hl `syncTick`, one show per step) and the
  5. // command-line tool (tools/sync-tmdb.hl). Reference for the API use: the old tracker's
  6. // crates/sync/commands/sync.js (read-only).
  7. //
  8. // GET <TMDB_BASE_URL>/tv/<tmdbId>?append_to_response=external_ids,aggregate_credits the details (status,
  9. // counts, overview, poster_path, seasons[])
  10. // + external_ids (imdb_id, tvdb_id) + the cast/crew
  11. // of every season (tracker#28: the answer goes back
  12. // as `res.details`, the caller stores the credits —
  13. // credits.hl applyCredits; no extra request)
  14. // GET <TMDB_BASE_URL>/tv/<tmdbId>?append_to_response=season/1,season/2… up to 20 seasons with their
  15. // episodes in ONE request
  16. // GET <TMDB_IMAGE_URL>/w342<poster_path> the poster → storage/mpackdb/posters/
  17. // Authorization: Bearer $TMDB_READ_TOKEN
  18. // GET <TVMAZE_BASE_URL>/lookup/shows?imdb=tt… (else ?thetvdb=<id>) the TVmaze id, ONLY when the show
  19. // has none (301 → /shows/<id>, 404 = unknown)
  20. // GET <TVMAZE_BASE_URL>/shows/<tvmzId>?embed=episodes a SERIES on TVmaze with its episodes (#15), see
  21. // below; its summary → `tvmazeSummary` (tracker#18)
  22. // A MOVIE (`type = 'movie'`, 7814 of the migrated rows, 1 followed): only
  23. // GET <TMDB_BASE_URL>/movie/<tmdbId>?append_to_response=external_ids,credits its imdb id + poster (+ credits); no seasons,
  24. // status, overview (it is no TV show)
  25. //
  26. // EXTERNAL IDS (tracker.worldapi.org#12, the show page's link icons): a MISSING imdbId/tvdbId is filled from TMDB's
  27. // external_ids, a missing tvmzId from TVmaze's lookup — an id the show already has is NEVER overwritten.
  28. // RULES (ticket): new seasons/episodes are ADDED, existing ones UPDATED (title, summary, release) —
  29. // matched by tmdbId, never duplicated; NOTHING is deleted. An episode without a tmdbId (161 migrated ones)
  30. // is matched by its S/E number instead and gets the tmdbId. A season (no tmdbId in the migrated data) is
  31. // matched by (show, seasonNumber). A TMDB value that is empty/null never overwrites a stored one.
  32. // Season 0 ("Specials") is only synced for a show that already has a season 0 (else every followed show
  33. // would suddenly flood /my/unwatched with specials); a TMDB season with no episodes is not created.
  34. //
  35. // TVMAZE MERGE (tracker.worldapi.org#15): AFTER TMDB, a series with a tvmzId gets TVmaze's episode list (one request)
  36. // and TVmaze FILLS GAPS only — matched by season/episode number: a missing episode (and its season) is ADDED, an empty
  37. // title or air date of an existing one is FILLED; a value TMDB gave is never overwritten. A placeholder title
  38. // ("Episode 4", "TBA") counts as empty — and TMDB's own placeholder never overwrites a real title either (else the
  39. // two would flip it every day). When TMDB later lists the episode, the TMDB step adopts it by number (it has no tmdbId)
  40. // and TMDB's values win. TVmaze specials (no episode number) are skipped, season 0 only if the show has one. SAFETY:
  41. // when the two number a show differently (fewer than 80 % of the episodes both know air within a day of each other or
  42. // share the title — `numberingAgrees`), TVmaze is ignored for that show (`tvmazeSkipped`). Summaries are not taken
  43. // (TVmaze's are HTML; the ticket asks for episodes, air dates, titles). The SHOW's TVmaze summary is kept, HTML stripped, as
  44. // `tvmazeSummary` (tracker#18; the page shows summary > tmdbSummary > tvmazeSummary, shows.hl summaryOf).
  45. //
  46. // WHICH TITLES, WHEN (tracker#18): the daily run asks TMDB's and TVmaze's change lists first and syncs only the titles of ours
  47. // that changed — dailysync.hl. THE SHOW RECORD (title, tagline, homepage, status, genres …) is refreshed from the same details
  48. // answer by details.hl `applySynced`.
  49. //
  50. // IDS ARE SELF-ASSIGNED (randomBytes, like watches.hl): hl:mpackdb's own generated @id can repeat within
  51. // one millisecond, which is what made update() lose rows (hybriel#113, fixed upstream, not in the vendored
  52. // build) — a table holding only unique ids is safe to update().
  53. //
  54. // ONE PROCESS PER STORAGE: hl:mpackdb does not coordinate two processes (hybriel#21) — the in-app run uses
  55. // the app's own open tables (imported from shows.hl); the tool only ever runs against a stopped app or a copy.
  56. import { now } from 'hl:time'
  57. import { newId, textOr } from './util.hl'
  58. import { showsTable, seasonsTable, episodesTable, kindOfTitle } from './shows.hl'
  59. import { tmdbGet, numIdOr, externalIdsFor, tvmazeEpisodes, syncPoster, adultOf } from './tmdb.hl'
  60. import { isPlaceholderTitle, numberingAgrees, wantedSeasonNumbers, batchesOf, changedText, isMigratedRow } from './sync-helpers.hl'
  61. // Merge TVmaze's episodes of a series into what we have AFTER the TMDB step (gaps only, rules at the top).
  62. // `seasonsByNumber` / `epByNumber` = the show's seasons by number and episodes by '<s>x<e>', as the TMDB step left them.
  63. // Answers { tvmazeRequests, newSeasons, newEpisodes, filledEpisodes, skipped, errors, error, showSeasons }.
  64. static mergeTvmaze = (title, &showKey, tvmzId, seasonsIn, episodesIn, &showSeasonsIn) => {
  65. seasonsByNumber = seasonsIn
  66. epByNumber = episodesIn
  67. out = { tvmazeRequests = 1 newSeasons = 0 newEpisodes = 0 filledEpisodes = 0 skipped = 0 errors = 0 error = '' showSeasons = showSeasonsIn summary = null }
  68. got = tvmazeEpisodes(tvmzId)
  69. if (got.failed != null) { out.errors = 1 out.error = title + ': ' + got.failed return out }
  70. out.summary = got.summary
  71. // the numbered episodes TVmaze has in seasons we take (no specials; season 0 only when the show has one)
  72. list = []
  73. for (te of got.list) {
  74. if (numIdOr(te.season) != null && numIdOr(te.number) != null) { list.push(te) }
  75. else if (te.season == 0 && numIdOr(te.number) != null && seasonsByNumber['0'] != null) { list.push(te) }
  76. }
  77. if (!numberingAgrees(epByNumber, list).ok) { out.skipped = 1 return out }
  78. added = {}
  79. addedFirstDate = {}
  80. for (te of list) {
  81. let key = te.season + 'x' + te.number
  82. let e = epByNumber[key]
  83. if (e == null) {
  84. let seasonKey = seasonsByNumber['' + te.season] != null ? seasonsByNumber['' + te.season].id : null
  85. if (seasonKey == null) {
  86. if (added['' + te.season] == null) { added['' + te.season] = [] addedFirstDate['' + te.season] = textOr(te.airdate) seasonsByNumber['' + te.season] = { id = newId() isNew = true } }
  87. seasonKey = seasonsByNumber['' + te.season].id
  88. }
  89. let fresh = { id = newId() episodeNumber = te.number seasonNumber = te.season
  90. title = textOr(te.name) summary = '' release = textOr(te.airdate)
  91. show = showKey season = seasonKey tmdbId = null tvmzId = te.id oldId = 'tvmaze-episode:' + te.id }
  92. if (episodesTable.put(fresh) == null) {
  93. out.errors = out.errors + 1
  94. out.error = title + ': TVmaze episode ' + te.id + ' not stored: ' + episodesTable.lastError()
  95. } else {
  96. if (added['' + te.season] == null) { added['' + te.season] = [] }
  97. let l = added['' + te.season]
  98. l.push(fresh.id)
  99. added['' + te.season] = l
  100. epByNumber[key] = fresh
  101. out.newEpisodes = out.newEpisodes + 1
  102. }
  103. } else {
  104. let changed = false
  105. if (isPlaceholderTitle(e.title) && !isPlaceholderTitle(te.name)) { e.title = te.name changed = true }
  106. if (textOr(e.release) == null && textOr(te.airdate) != null) { e.release = te.airdate changed = true }
  107. if (changed) {
  108. if (episodesTable.update(e.id, e) == null) {
  109. out.errors = out.errors + 1
  110. out.error = title + ': episode ' + e.id + ' not filled from TVmaze: ' + episodesTable.lastError()
  111. } else {
  112. epByNumber[key] = e
  113. out.filledEpisodes = out.filledEpisodes + 1
  114. }
  115. }
  116. }
  117. }
  118. // the seasons that got episodes: an existing one is updated, a new one stored and linked to the show
  119. showSeasons = []
  120. for (sid of showSeasonsIn) { showSeasons.push(sid) }
  121. for (n of added.keys()) {
  122. let ids = added[n]
  123. let s = seasonsByNumber[n]
  124. if (ids.length > 0 && s.isNew == true) {
  125. let season = { id = s.id seasonNumber = toNumber(n) title = 'Season ' + n release = addedFirstDate[n] summary = ''
  126. show = showKey episodes = ids oldId = 'tvmaze-season:' + tvmzId + ':' + n }
  127. if (seasonsTable.put(season) == null) {
  128. out.errors = out.errors + 1
  129. out.error = title + ': TVmaze season ' + n + ' not stored: ' + seasonsTable.lastError()
  130. } else {
  131. showSeasons.push(season.id)
  132. out.newSeasons = out.newSeasons + 1
  133. }
  134. } else if (ids.length > 0) {
  135. let season = seasonsTable.fetch(s.id)
  136. let eps = []
  137. for (eid of season.episodes) { eps.push(eid) }
  138. for (eid of ids) { eps.push(eid) }
  139. season.episodes = eps
  140. if (seasonsTable.update(season.id, season) == null) {
  141. out.errors = out.errors + 1
  142. out.error = title + ': season ' + season.id + ' not updated: ' + seasonsTable.lastError()
  143. }
  144. }
  145. }
  146. out.showSeasons = showSeasons
  147. return out
  148. }
  149. // a MOVIE: one request — its imdb id (if missing) and its poster; nothing else of the record is touched. `pre` = that
  150. // request's answer when the caller already made it (details.hl firstStep, tracker#26), else null
  151. static syncMovie = (show, res, &pre) => {
  152. out = res
  153. out.requests = 1
  154. details = pre != null ? pre : tmdbGet('/movie/' + show.tmdbId + '?append_to_response=external_ids,credits')
  155. if (details.failed != null) { out.errors = 1 out.error = show.title + ': ' + details.failed return out }
  156. if (pre == null) { out.details = details }
  157. ids = externalIdsFor(show, details.external_ids, false, false)
  158. out.ids = ids.filled
  159. poster = syncPoster(show, details.poster_path)
  160. out.requests = out.requests + poster.requests
  161. out.posters = poster.saved
  162. if (poster.failed != null) { out.errors = out.errors + 1 out.error = show.title + ': ' + poster.failed }
  163. m = show
  164. m.imdbId = ids.imdbId
  165. m.image = poster.image
  166. m.tmdbPoster = poster.tmdbPoster
  167. if (adultOf(details) != null) { m.adult = adultOf(details) }
  168. m.tmdbSync = now()
  169. if (showsTable.update(m.id, m) == null) { out.errors = out.errors + 1 out.error = show.title + ': movie not updated: ' + showsTable.lastError() }
  170. return out
  171. }
  172. // Sync ONE show (by our id). Answers counts: { requests, tvmazeRequests, newSeasons, newEpisodes, updatedEpisodes,
  173. // updatedSeasons, posters, ids, tvmazeEpisodes, tvmazeSeasons, tvmazeFilled, tvmazeSkipped, errors, error } + `details` (the
  174. // details answer with its credits when this call asked for it, else null — tracker#28: the caller stores the credits). Never throws for a TMDB failure — it is counted and the show left as is.
  175. static syncShow = (&showId) => { return syncShowPart(showId, null, null) }
  176. // the same with the details request (`/tv|movie/<id>?append_to_response=external_ids…`) already answered — `pre`, counted as
  177. // this show's first request — so details.hl (tracker#26) asks TMDB once for details, ids AND the cast
  178. static syncShowWith = (&showId, &pre) => { return syncShowPart(showId, pre, null) }
  179. // ONE PART of a series' sync (tracker#26: a talk show with 4,000 episodes took 10 s in one step, blocking every request):
  180. // `only` = { seasons (the TMDB season numbers of this part), finish (false: only those seasons — new ones linked to the show,
  181. // nothing else; true: the ids, TVmaze, poster and the show record), tvmaze (false: no TVmaze lookup/merge — tracker#18's light
  182. // step) }; null = everything at once. `pre` is required with `only` (its request is the caller's: a part counts only its own
  183. // requests).
  184. static emptyResult = () => { return { requests = 0 tvmazeRequests = 0 newSeasons = 0 newEpisodes = 0 updatedEpisodes = 0 updatedSeasons = 0 posters = 0 ids = 0 tvmazeEpisodes = 0 tvmazeSeasons = 0 tvmazeFilled = 0 tvmazeSkipped = 0 errors = 0 error = '' details = null } }
  185. static syncShowPart = (&showId, &pre, &only) => {
  186. let res = emptyResult()
  187. show = showsTable.fetch(showId)
  188. if (show == null) { res.errors = 1 res.error = 'no show ' + showId return res }
  189. if (show.tmdbId == null) { res.errors = 1 res.error = show.title + ': no tmdbId' return res }
  190. if (show.type == 'movie') { return syncMovie(show, res, pre) }
  191. res.requests = only != null ? 0 : 1
  192. details = pre != null ? pre : tmdbGet('/tv/' + show.tmdbId + '?append_to_response=external_ids,aggregate_credits')
  193. if (details.failed != null) { res.errors = 1 res.error = show.title + ': ' + details.failed return res }
  194. if (pre == null) { res.details = details }
  195. // what we have: seasons by number, episodes by tmdbId and by S/E number
  196. // (tracker#18: a part reads only ITS seasons' episodes, unless it finishes with the TVmaze merge, which compares them all —
  197. // Today, 15,716 episodes, read them all for a one-season change: 20 s and GBs of memory)
  198. tvmazeToo = only == null || only.tvmaze != false
  199. allEpisodes = only == null || (only.finish && tvmazeToo)
  200. seasonsByNumber = {}
  201. epByTmdb = {}
  202. epByNumber = {}
  203. for (sid of show.seasons) {
  204. let s = seasonsTable.fetch(sid)
  205. if (s != null) {
  206. seasonsByNumber['' + s.seasonNumber] = s
  207. let epIds = allEpisodes || only.seasons.includes(s.seasonNumber) ? s.episodes : []
  208. for (eid of epIds) {
  209. let e = episodesTable.fetch(eid)
  210. if (e != null) {
  211. if (e.tmdbId != null) { epByTmdb['' + e.tmdbId] = e }
  212. epByNumber[e.seasonNumber + 'x' + e.episodeNumber] = e
  213. }
  214. }
  215. }
  216. }
  217. // plain locals for the record literals below: inside a literal, an entry named `show`/`season` would shadow
  218. // the variable for every later entry (SPEC "A literal's entries are evaluated in order")
  219. showKey = show.id
  220. showTmdb = show.tmdbId
  221. let showSeasons = []
  222. for (sid of show.seasons) { showSeasons.push(sid) }
  223. let showChanged = false
  224. let wanted = wantedSeasonNumbers(details, seasonsByNumber)
  225. if (only != null) {
  226. part = []
  227. for (n of wanted) { if (only.seasons.includes(n)) { part.push(n) } }
  228. wanted = part
  229. }
  230. for (batch of batchesOf(wanted, 20)) {
  231. let names = []
  232. for (n of batch) { names.push('season/' + n) }
  233. res.requests = res.requests + 1
  234. let withSeasons = tmdbGet('/tv/' + show.tmdbId + '?append_to_response=' + names.join(','))
  235. if (withSeasons.failed != null) {
  236. res.errors = res.errors + 1
  237. res.error = show.title + ': ' + withSeasons.failed
  238. } else {
  239. for (n of batch) {
  240. let ts = withSeasons['season/' + n]
  241. if (ts != null && ts.episodes != null) {
  242. let season = seasonsByNumber['' + n]
  243. let seasonIsNew = season == null
  244. let seasonChanged = false
  245. if (seasonIsNew) {
  246. season = { id = newId() seasonNumber = n title = textOr(ts.name) != null ? ts.name : 'Season ' + n
  247. release = textOr(ts.air_date) summary = textOr(ts.overview) != null ? ts.overview : ''
  248. show = showKey episodes = [] oldId = 'tmdb-season:' + showTmdb + ':' + n }
  249. } else {
  250. let seasonMigrated = isMigratedRow(season)
  251. if (changedText(season.title, textOr(ts.name))) {
  252. if (seasonMigrated && season.migratedTitle == null && textOr(season.title) != null) { season.migratedTitle = season.title }
  253. season.title = ts.name seasonChanged = true
  254. }
  255. if (changedText(season.summary, textOr(ts.overview))) {
  256. if (seasonMigrated && season.migratedSummary == null && textOr(season.summary) != null) { season.migratedSummary = season.summary }
  257. season.summary = ts.overview seasonChanged = true
  258. }
  259. if (changedText(season.release, textOr(ts.air_date))) { season.release = ts.air_date seasonChanged = true }
  260. }
  261. let seasonKey = season.id
  262. let seasonEpisodes = []
  263. for (eid of season.episodes) { seasonEpisodes.push(eid) }
  264. for (te of ts.episodes) {
  265. if (te.id != null && te.episode_number != null) {
  266. let e = epByTmdb['' + te.id]
  267. if (e == null) {
  268. let byNum = epByNumber[n + 'x' + te.episode_number]
  269. if (byNum != null && byNum.tmdbId == null) { e = byNum }
  270. }
  271. if (e == null) {
  272. let fresh = { id = newId() episodeNumber = te.episode_number seasonNumber = n
  273. title = textOr(te.name) summary = textOr(te.overview) != null ? te.overview : '' release = textOr(te.air_date)
  274. show = showKey season = seasonKey tmdbId = te.id oldId = 'tmdb-episode:' + te.id }
  275. let putId = episodesTable.put(fresh)
  276. if (putId == null) {
  277. res.errors = res.errors + 1
  278. res.error = show.title + ': episode ' + te.id + ' not stored: ' + episodesTable.lastError()
  279. } else {
  280. seasonEpisodes.push(fresh.id)
  281. epByTmdb['' + te.id] = fresh
  282. epByNumber[n + 'x' + te.episode_number] = fresh
  283. res.newEpisodes = res.newEpisodes + 1
  284. seasonChanged = true
  285. }
  286. } else {
  287. let changed = false
  288. if (e.tmdbId == null) { e.tmdbId = te.id changed = true }
  289. // #15: TMDB's "Episode 4" never replaces a real title (one TVmaze filled, or the migrated one)
  290. let epMigrated = isMigratedRow(e)
  291. if (changedText(e.title, textOr(te.name)) && !(isPlaceholderTitle(te.name) && !isPlaceholderTitle(e.title))) {
  292. if (epMigrated && e.migratedTitle == null && textOr(e.title) != null) { e.migratedTitle = e.title }
  293. e.title = te.name changed = true
  294. }
  295. if (changedText(e.summary, textOr(te.overview))) {
  296. if (epMigrated && e.migratedSummary == null && textOr(e.summary) != null) { e.migratedSummary = e.summary }
  297. e.summary = te.overview changed = true
  298. }
  299. if (changedText(e.release, textOr(te.air_date))) { e.release = te.air_date changed = true }
  300. if (changed) {
  301. if (episodesTable.update(e.id, e) == null) {
  302. res.errors = res.errors + 1
  303. res.error = show.title + ': episode ' + e.id + ' not updated: ' + episodesTable.lastError()
  304. } else {
  305. epByTmdb['' + e.tmdbId] = e
  306. epByNumber[e.seasonNumber + 'x' + e.episodeNumber] = e
  307. res.updatedEpisodes = res.updatedEpisodes + 1
  308. }
  309. }
  310. }
  311. }
  312. }
  313. season.episodes = seasonEpisodes
  314. if (seasonIsNew) {
  315. if (seasonsTable.put(season) == null) {
  316. res.errors = res.errors + 1
  317. res.error = show.title + ': season ' + n + ' not stored: ' + seasonsTable.lastError()
  318. } else {
  319. seasonsByNumber['' + n] = season
  320. showSeasons.push(season.id)
  321. showChanged = true
  322. res.newSeasons = res.newSeasons + 1
  323. }
  324. } else if (seasonChanged) {
  325. if (seasonsTable.update(season.id, season) == null) {
  326. res.errors = res.errors + 1
  327. res.error = show.title + ': season ' + season.id + ' not updated: ' + seasonsTable.lastError()
  328. } else {
  329. res.updatedSeasons = res.updatedSeasons + 1
  330. }
  331. }
  332. }
  333. }
  334. }
  335. }
  336. // a part that is not the last: only the new seasons are linked to the show (the next part reads them)
  337. if (only != null && !only.finish) {
  338. if (showChanged) {
  339. show.seasons = showSeasons
  340. if (showsTable.update(show.id, show) == null) { res.errors = res.errors + 1 res.error = show.title + ': show not updated: ' + showsTable.lastError() }
  341. }
  342. return res
  343. }
  344. // the external ids (tracker.worldapi.org#12): missing ones from TMDB's external_ids, a missing TVmaze id from TVmaze
  345. ids = externalIdsFor(show, details.external_ids, true, tvmazeToo)
  346. res.ids = ids.filled
  347. res.tvmazeRequests = ids.tvmazeRequests
  348. if (ids.failed != null) { res.errors = res.errors + 1 res.error = show.title + ': ' + ids.failed }
  349. // tracker.worldapi.org#15: what TVmaze has and TMDB not yet (new episodes/seasons, empty titles and air dates)
  350. let tvmazeSummary = null
  351. if (numIdOr(ids.tvmzId) != null && tvmazeToo) {
  352. tvm = mergeTvmaze(show.title, showKey, ids.tvmzId, seasonsByNumber, epByNumber, showSeasons)
  353. res.tvmazeRequests = res.tvmazeRequests + tvm.tvmazeRequests
  354. res.tvmazeEpisodes = tvm.newEpisodes
  355. res.tvmazeSeasons = tvm.newSeasons
  356. res.tvmazeFilled = tvm.filledEpisodes
  357. res.tvmazeSkipped = tvm.skipped
  358. if (tvm.errors > 0) { res.errors = res.errors + tvm.errors res.error = tvm.error }
  359. showSeasons = tvm.showSeasons
  360. tvmazeSummary = tvm.summary
  361. }
  362. // the show itself: status, counts, TMDB's overview, the poster, when it was synced
  363. poster = syncPoster(show, details.poster_path)
  364. res.requests = res.requests + poster.requests
  365. res.posters = poster.saved
  366. if (poster.failed != null) { res.errors = res.errors + 1 res.error = show.title + ': ' + poster.failed }
  367. show.seasons = showSeasons
  368. if (textOr(details.status) != null) { show.status = details.status }
  369. if (details.number_of_seasons != null) { show.seasonsCount = details.number_of_seasons }
  370. if (details.number_of_episodes != null) { show.episodesCount = details.number_of_episodes }
  371. // tracker#18: TMDB's text → tmdbSummary, TVmaze's → tvmazeSummary; `summary` is the creator's own text, never written here
  372. if (textOr(details.overview) != null) { show.tmdbSummary = details.overview }
  373. if (tvmazeSummary != null) { show.tvmazeSummary = tvmazeSummary }
  374. show.image = poster.image
  375. show.tmdbPoster = poster.tmdbPoster
  376. show.imdbId = ids.imdbId
  377. show.tvdbId = ids.tvdbId
  378. show.tvmzId = ids.tvmzId
  379. if (adultOf(details) != null) { show.adult = adultOf(details) }
  380. if (textOr(details.type) != null) { show.tmdbType = details.type show.kind = kindOfTitle(details.type, show.genres) }
  381. show.tmdbSync = now()
  382. if (showsTable.update(show.id, show) == null) { res.errors = res.errors + 1 res.error = show.title + ': show not updated: ' + showsTable.lastError() }
  383. return res
  384. }
  385. // ---- the LIGHT step of the daily delta run (tracker#18) ---------------------------------------------------------------
  386. // A changed title NOBODY follows (dailysync.hl): its details (+ ids + credits: the record, applySynced) and only the seasons
  387. // that can have changed — the ones we don't have, the ones where TMDB counts MORE episodes than we have (fewer: TVmaze's
  388. // extras, nothing to fetch), and the newest one (new or renamed episodes land there). No TVmaze (its merge only matters for the followed shows' lists; their full step still does
  389. // it). Real data (2026-10-01): Today (67 seasons / 15,716 episodes) and EastEnders (43 / 7,349) change every day; their full
  390. // sync blocked the server 20.8 s each and grew the process by ~5 GB each.
  391. static changedSeasonNumbers = (&details, &show) => {
  392. ours = {}
  393. for (sid of show.seasons) {
  394. let s = seasonsTable.fetch(sid)
  395. if (s != null) { ours['' + s.seasonNumber] = s.episodes.length }
  396. }
  397. let latest = -1
  398. for (x of (details.seasons != null ? details.seasons : [])) {
  399. if (x != null && x.season_number != null && x.season_number > latest && (x.episode_count == null || x.episode_count > 0)) { latest = x.season_number }
  400. }
  401. out = []
  402. for (x of (details.seasons != null ? details.seasons : [])) {
  403. if (x != null && x.season_number != null) {
  404. let n = x.season_number
  405. if (n == latest || ours['' + n] == null || (x.episode_count != null && x.episode_count > ours['' + n])) { out.push(n) }
  406. }
  407. }
  408. return out
  409. }
  410. static syncShowDelta = (&showId) => {
  411. show = showsTable.fetch(showId)
  412. if (show == null || show.tmdbId == null || show.type == 'movie') { return syncShow(showId) }
  413. details = tmdbGet('/tv/' + show.tmdbId + '?append_to_response=external_ids,aggregate_credits')
  414. if (details.failed != null) {
  415. let res = emptyResult()
  416. res.requests = 1
  417. res.errors = 1
  418. res.error = show.title + ': ' + details.failed
  419. return res
  420. }
  421. r = syncShowPart(showId, details, { seasons = changedSeasonNumbers(details, show) finish = true tvmaze = false })
  422. r.requests = r.requests + 1
  423. r.details = details
  424. return r
  425. }
  426. // flush the three tables' index/meta state to disk (after a run, and every few shows)
  427. static persistSync = () => {
  428. showsTable.persist()
  429. seasonsTable.persist()
  430. episodesTable.persist()
  431. return true
  432. }

Branches

Latest commits

  • 7d7d4487tracker#33 (mission 029): reduced titles — every title TMDB's details never went through this app (no detailsAt, no tmdbSync) is incomplete (shows.hl isIncomplete; the old tracker's migrated rows passed #26's test: 5,697 non-adult on the live copy, 691 series without seasons); the repair job does the visibly reduced first (shows.hl missingParts), the page completes one on open; a title TMDB has no poster for (The Remaining) shows the placeholder; tools/count-incomplete.hl; gate fixtures stand for synced titles (tmdbSync), tests/seed-reduced.hl + #33 checks; gates 347/0, 32/0, 52/0mre
  • 661c2592tracker: report 028mre
  • 27c916fatracker mission 028: README ("Code order", the new file map), STATUS (counts before/after, tests, how to repeat, open), LOGmre
  • d924f398tracker mission 028: comments name the new files (sync.hl, dailysync.hl, backfill.hl, credits.hl, jobs.hl, images.hl …); tools/ref-params.py + tools/lambda-audit.py also scan lib/ (they globbed the root only), lambda-audit counts a plain `x = p` alias like `let x = p`mre
  • 2e89b968tracker mission 028 (code order) 5/5 let: `let` only where a variable is reassigned — 667 never-reassigned lets became plain declarations (project.hl, lib/, components/, tools/, tests/); kept: 264 in loop bodies (a plain declaration there is 'Cannot reassign' on the 2nd pass), 234 reassigned, 27 whose name is also a member/outer/free name (a plain write would rebind it); tools/let-audit.py decides and fixes (README 'Code order'); tests/realdata-m028.{sh,mjs} = the page-output diff on a real copy; gates 342/0, 32/0, 52/0, real-copy pages identicalmre
  • 54796ff2tracker mission 028 (code order) 4/5 thin faces + last copies: the show page's check/follow faces call lib/watches.hl toggleWatched / toggleSeasonWatched (seasonAllWatched moved there) and lib/follows.hl toggleFollowed; both logins (header selector face, /login/callback) share lib/users.hl userOfCode; todayStr/listOf copies in components and the export readers copied into tools/migrate.hl + tools/old-short-ids.hl now once (lib/util.hl, lib/export.hl); gates 342/0, 32/0, 52/0; old-short-ids output byte-identical, migrate output identicalmre
  • 06b078e3tracker mission 028 (code order) 3/5 project.hl is the map: config, routes, wiring and a feature → file index (914 → 258 lines); the background jobs (daily sync run, backfills, details repair, credits job, merge, short ids, collection seed) moved unchanged into lib/jobs.hl (a class: their state is reassigned every step, a static cannot be; one instance made after the server), the login callback into lib/users.hl, poster/photo serving into lib/images.hl, the /shows/<slug> rule into lib/shows.hl showsMovedPath; route handlers are thin wrappers; gates 342/0, 32/0, 52/0, real-copy pages identicalmre
  • 94716fd2tracker mission 028 (code order) 2/5 util + topics: lib/util.hl holds envOr, storageDir, postersDir, profilesDir, newId, hexDigits, todayStr, dateOr, textOr, hasId, listOr, firstOf, sortDesc once (were copied into up to 5 files); tmdbsync.hl split into tmdb.hl (TMDB/TVmaze requests), sync.hl (one title's sync), sync-helpers.hl, backfill.hl; details.hl split into details.hl, credits.hl, credits-helpers.hl (isIncomplete to shows.hl); search-helpers.hl (words, query, ranking, slugs); collections.hl (the TMDB collection seed, out of franchises.hl); deltasync.hl renamed dailysync.hl; no behaviour change: gates 342/0, 32/0, 52/0, real-copy pages identicalmre
  • 186079b0tracker mission 028 (code order) 1/5 move: every root .hl except project.hl into lib/ (styles.hl into components/), import paths only; gates 342/0, 32/0, 52/0; real-copy pages identicalmre
  • 4f47f181tracker: report 027mre
  • dc1d4be4tracker mission 027: Hybriel master 06617221 vendored (plugin allocator fixes 3a781359 + 413f60e4); real copy RSS through first-start jobs + 400 loads flat ~2.55 GB (190aa11d 2.3 -> 5.6 GB), page times <= 1.1x; gates 342/0, 32/0, 52/0mre
  • 84e1b3e1tracker: reports 025 + 026mre
  • dc40d859tracker#31 (mission 026): duplicate titles merged — the 68 type+tmdbId pairs held by 157 records were the old tracker's (all migrated); merge.hl repair job (own clock, before the TMDB jobs) keeps one keeper per title (follows/watches > old short id > oldest), moves follows, watches, seasons, cast, credits, timelines, tombstones the rest (mergedInto, never deleted), slugs + short ids 301 to the keeper; stray seasons merged into their listed twin (Reacher S3 watches) or linked when watched; search import re-checks before its put; deploy.sh waits up to 90 s for 200; real copy 68 -> 0 dup ids, az5b2 follows/watches equal; gates 342/0, 32/0, 52/0mre
  • 2667da05tracker#30 (mission 025): /my/ pages from slim cached title cards, episode rows and watch sets (after the jobs /my/series 1.8 s -> 0.06 s, /my/unwatched 4.1 -> 0.18 s); timeline page shows its name once; franchise widget under the poster/title; movies with TV leftovers (First Contact) go through the details repair; tools/count-tmdb-ids.hl; gates 327/0, 52/0, 32/0mre
  • 3909810dantcolony#40: mission references in README/STATUS/docs point to the moved missionsmre
  • e63b1d28antcolony#40: history (LOG.md), worker briefs (missions/) and reports moved here from antcolony, numbered per project; old numbers in antcolony docs/mission-map.mdmre
  • 1cda451dtracker: Hybriel master 190aa11d (#127 both shapes, GC correctness fc838894) — conductor adopts despite /my/series 2.4x after jobs (memory 8.0 → 1.7 GB boot); see reports/071mre
  • c1fa2f2etracker (mission 071): Hybriel master 190aa11d measured on the real copy vs the live binary 8590df63 — NOT adopted (after the first-start jobs /my/series 2.4x slower, /series 1.6x, RSS swings 7.4-12.2 GB; fresh it is flat at 1.7-2.2 GB and /my/unwatched faster), vendor stays 8efba065, candidate kept in .scratch/w071/vendor-190aa11d; tests/kinds.mjs: collection seed off (its TMDB request broke check 1 in 1 of 4 runs); tests/realdata-071.sh + realdata-071-bench.mjs + tools/realdata-071-table.py; README + STATUS (numbers, how to repeat); gates 325/0, 32/0, 50/0mre
  • 8081350atracker docs (mission 070): README (summary, Config HL_GC_BYTES — kept at Hybriel's default, the 256 MiB setting is taken out of docker-compose.yml again: the jobs grew to 12+ GB with it too, see STATUS), Test (three gates), Deploy (first start ~50 min: kinds then seed, restart once after collections done, memory numbers), Vendored Hybriel 8efba065 + #48 audit, Files; STATUS mission 070 entry (merges, migrated counts, lambda audit, gates, RSS old vs new, how to repeat, open points); docs/kinds.md + docs/franchises.md job order; tests/realdata-070-*.sh, tools/count-migrated.hl, tools/ref-params.py, tools/lambda-audit.pymre
  • 1ad19c8ctracker: re-vendor Hybriel master 8efba065 (#126 GC by bytes, #48 lambda parameters copy) (mission 070): bin/hybriel sha256 50361e95…, plugins core crypto data fetch fs http http1 mpackdb proc smtp time web; lambda audit: 13 lambdas change a passed record/list (11 through a local alias), no caller relies on it — unchanged; 319 read-only lambda parameters get & (no copy per call: /my/schedule 2.35 → 0.46 s, /my/unwatched 13.7 → 5.7 s on the real copy); /my/unwatched one merge sort instead of n² inserts; docker-compose HL_GC_BYTES=268435456; deploy.sh runs kinds.mjs + franchises.mjs too (default ports 8700–8710); tests/realdata-070.mjs; gates browser 325/0, kinds 32/0, franchises 50/0mre