gitoriaLog in with ident

tracker

All repositories: gitoria

ReadmeCodePull requestsReleasesTicketsSettings
Commit46b2138946b21389tracker (mission 070, conductor): the sync never destroys migrated data — the one-time summary step MOVES a copied summary to migratedSummary (marker summaries-moved.txt) instead of clearing it; a migrated record's first title/genres/homepage/tagline TMDB replaces → migratedTitle/migratedGenres/migratedHomepage/migratedTagline, a migrated season's/episode's title/summary → migratedTitle/migratedSummary (set once); tests/peek-shows.hl prints them; gate 325/0mre46b21389/tmdbsync.hl

45.1 KB

  1. // tmdbsync.hl — THE TMDB SYNC (tracker.worldapi.org#9): for every show SOMEONE follows, fetch the show's
  2. // details, seasons and episodes from TMDB and bring storage/mpackdb/ up to date, plus the show's poster.
  3. // Used by BOTH runners: the daily run inside the app (project.hl `syncTick`, one show per step) and the
  4. // command-line tool (tools/sync-tmdb.hl). Reference for the API use: the old tracker's
  5. // crates/sync/commands/sync.js (read-only).
  6. //
  7. // GET <TMDB_BASE_URL>/tv/<tmdbId>?append_to_response=external_ids,aggregate_credits the details (status,
  8. // counts, overview, poster_path, seasons[])
  9. // + external_ids (imdb_id, tvdb_id) + the cast/crew
  10. // of every season (tracker#28: the answer goes back
  11. // as `res.details`, the caller stores the credits —
  12. // details.hl applyCredits; no extra request)
  13. // GET <TMDB_BASE_URL>/tv/<tmdbId>?append_to_response=season/1,season/2… up to 20 seasons with their
  14. // episodes in ONE request
  15. // GET <TMDB_IMAGE_URL>/w342<poster_path> the poster → storage/mpackdb/posters/
  16. // Authorization: Bearer $TMDB_READ_TOKEN
  17. // GET <TVMAZE_BASE_URL>/lookup/shows?imdb=tt… (else ?thetvdb=<id>) the TVmaze id, ONLY when the show
  18. // has none (301 → /shows/<id>, 404 = unknown)
  19. // GET <TVMAZE_BASE_URL>/shows/<tvmzId>?embed=episodes a SERIES on TVmaze with its episodes (#15), see
  20. // below; its summary → `tvmazeSummary` (tracker#18)
  21. // A MOVIE (`type = 'movie'`, 7814 of the migrated rows, 1 followed): only
  22. // GET <TMDB_BASE_URL>/movie/<tmdbId>?append_to_response=external_ids,credits its imdb id + poster (+ credits); no seasons,
  23. // status, overview (it is no TV show)
  24. //
  25. // EXTERNAL IDS (tracker.worldapi.org#12, the show page's link icons): a MISSING imdbId/tvdbId is filled from TMDB's
  26. // external_ids, a missing tvmzId from TVmaze's lookup — an id the show already has is NEVER overwritten.
  27. // RULES (ticket): new seasons/episodes are ADDED, existing ones UPDATED (title, summary, release) —
  28. // matched by tmdbId, never duplicated; NOTHING is deleted. An episode without a tmdbId (161 migrated ones)
  29. // is matched by its S/E number instead and gets the tmdbId. A season (no tmdbId in the migrated data) is
  30. // matched by (show, seasonNumber). A TMDB value that is empty/null never overwrites a stored one.
  31. // Season 0 ("Specials") is only synced for a show that already has a season 0 (else every followed show
  32. // would suddenly flood /my/unwatched with specials); a TMDB season with no episodes is not created.
  33. //
  34. // TVMAZE MERGE (tracker.worldapi.org#15): AFTER TMDB, a series with a tvmzId gets TVmaze's episode list (one request)
  35. // and TVmaze FILLS GAPS only — matched by season/episode number: a missing episode (and its season) is ADDED, an empty
  36. // title or air date of an existing one is FILLED; a value TMDB gave is never overwritten. A placeholder title
  37. // ("Episode 4", "TBA") counts as empty — and TMDB's own placeholder never overwrites a real title either (else the
  38. // two would flip it every day). When TMDB later lists the episode, the TMDB step adopts it by number (it has no tmdbId)
  39. // and TMDB's values win. TVmaze specials (no episode number) are skipped, season 0 only if the show has one. SAFETY:
  40. // when the two number a show differently (fewer than 80 % of the episodes both know air within a day of each other or
  41. // share the title — `numberingAgrees`), TVmaze is ignored for that show (`tvmazeSkipped`). Summaries are not taken
  42. // (TVmaze's are HTML; the ticket asks for episodes, air dates, titles). The SHOW's TVmaze summary is kept, HTML stripped, as
  43. // `tvmazeSummary` (tracker#18; the page shows summary > tmdbSummary > tvmazeSummary, shows.hl summaryOf).
  44. //
  45. // WHICH TITLES, WHEN (tracker#18): the daily run asks TMDB's and TVmaze's change lists first and syncs only the titles of ours
  46. // that changed — deltasync.hl. THE SHOW RECORD (title, tagline, homepage, status, genres …) is refreshed from the same details
  47. // answer by details.hl `applySynced`.
  48. //
  49. // ADULT FLAG (mission 054): every details answer carries TMDB's `adult` — the sync, the import (search.hl) and the
  50. // BACKFILL below store it as show.adult (true/false; null = not known yet, hidden from the public lists, shows.hl
  51. // isPublicTitle). THE BACKFILL (`backfillTitle`, run one title per step by project.hl `backfillTick`, like the daily sync):
  52. // GET <TMDB_BASE_URL>/movie|tv/<tmdbId>?append_to_response=external_ids ONE request: adult, poster_path, imdb/tvdb id
  53. // GET <TMDB_IMAGE_URL>/w342<poster_path> the poster when its file is missing (not for
  54. // an adult title — never shown, mission 056)
  55. // for every title whose adult flag is unknown (no TVmaze, no seasons — the daily sync does those for followed shows). A
  56. // title TMDB answers 404 for (or one without tmdbId) gets `adultCheck` = why, stays unknown (hidden) and is not asked again.
  57. //
  58. // KIND (tracker.worldapi.org#21, mission 068): a TV details answer carries TMDB's `type` (Scripted, Miniseries, Documentary,
  59. // Reality, Talk Show, News, Video) — the sync, the adult backfill and the import (search.hl) store it as show.tmdbType and the
  60. // title's kind ('series' | 'show', shows.hl kindOfTitle) as show.kind. THE KIND BACKFILL (`kindTitle`, run one title per step by
  61. // project.hl `kindTick`, after the adult backfill, on the same clock):
  62. // a title with a show genre (Reality / Talk / News) kind = 'show' from the stored data, NO request
  63. // any other TV title without `kind` GET <TMDB_BASE_URL>/tv/<tmdbId> ONE request → tmdbType + kind
  64. // A 404 (or no tmdbId) stores the genre's answer ('series') with `kindCheck` = why; it is not asked again. Resumes after a
  65. // restart (the queue = every TV title still without `kind`).
  66. //
  67. // IDS ARE SELF-ASSIGNED (randomBytes, like watches.hl): hl:mpackdb's own generated @id can repeat within
  68. // one millisecond, which is what made update() lose rows (hybriel#113, fixed upstream, not in the vendored
  69. // build) — a table holding only unique ids is safe to update().
  70. //
  71. // ONE PROCESS PER STORAGE: hl:mpackdb does not coordinate two processes (hybriel#21) — the in-app run uses
  72. // the app's own open tables (imported from shows.hl); the tool only ever runs against a stopped app or a copy.
  73. //
  74. // Env: TMDB_READ_TOKEN (required unless TMDB_BASE_URL points at a fake), TMDB_BASE_URL
  75. // (https://api.themoviedb.org/3), TMDB_IMAGE_URL (https://image.tmdb.org/t/p) — the gate points both at its
  76. // fake TMDB (tests/browser.mjs).
  77. import { fetch } from 'hl:fetch'
  78. import { env } from 'hl:proc'
  79. import { now } from 'hl:time'
  80. import { randomBytes } from 'hl:crypto'
  81. import { writeFile, mkDir, exists } from 'hl:fs'
  82. import { showsTable, seasonsTable, episodesTable, storageDir, kindOfTitle, hasShowGenre } from './shows.hl'
  83. static envOr = (name, fallback) => {
  84. let v = env(name)
  85. return v != null && v.trim() != '' ? v.trim() : fallback
  86. }
  87. static tmdbBase = envOr('TMDB_BASE_URL', 'https://api.themoviedb.org/3')
  88. static tmdbImageBase = envOr('TMDB_IMAGE_URL', 'https://image.tmdb.org/t/p')
  89. static tmdbToken = envOr('TMDB_READ_TOKEN', '')
  90. static postersDir = storageDir + '/posters'
  91. static tvmazeBase = envOr('TVMAZE_BASE_URL', 'https://api.tvmaze.com')
  92. // the sync can run: a token is set (never printed) — the dev .env on Loreana has one, a gate sets its own
  93. static syncEnabled = tmdbToken != ''
  94. // TMDB allows 40 requests / 10 s: every runner waits this long per request it made before the next show
  95. static msPerRequest = 260
  96. // TVmaze allows 20 requests / 10 s (its API docs): 550 ms per lookup made
  97. static msPerTvmazeRequest = 550
  98. // how long a runner waits after a show before the next one (both runners: project.hl syncTick, tools/sync-tmdb.hl)
  99. static pauseMsAfter = (r) => { return r.requests * msPerRequest + r.tvmazeRequests * msPerTvmazeRequest }
  100. // one GET → the parsed JSON, or { failed = '<why>' }. A refused/timed-out fetch is an Error event (absorbed
  101. // by the runner's `on Error`), the fetch then yields null.
  102. static tmdbGet = (path) => {
  103. let r = fetch(tmdbBase + path, { headers = { 'authorization' = 'Bearer ' + tmdbToken 'accept' = 'application/json' 'user-agent' = 'tracker.worldapi.org (tmdb sync)' } timeoutMs = 20000 })
  104. if (r == null || r.status == null || r.status == 0) { return { failed = 'no answer for ' + path } }
  105. if (r.status != 200) { return { failed = 'HTTP ' + r.status + ' for ' + path status = r.status } }
  106. let j = r.json()
  107. if (j == null) { return { failed = 'no JSON for ' + path } }
  108. return j
  109. }
  110. static textOr = (v) => { return v != null && hlTypeName(v) == 'String' && v.trim() != '' ? v : null }
  111. // ---- external ids (tracker.worldapi.org#12) -----------------------------------------------------
  112. static hasId = (v) => { return v != null && ('' + v).trim() != '' }
  113. // an IMDb id as TMDB gives it ('tt' + digits), else null
  114. static imdbOr = (v) => {
  115. let t = textOr(v)
  116. if (t == null || t.length < 3 || t.slice(0, 2) != 'tt') { return null }
  117. let i = 2
  118. while (i < t.length) {
  119. if (!'0123456789'.includes(t[i])) { return null }
  120. i = i + 1
  121. }
  122. return t
  123. }
  124. // a positive numeric id (TMDB's tvdb_id, TVmaze's id), else null
  125. static numIdOr = (v) => { return v != null && hlTypeName(v) == 'Number' && v > 0 ? v : null }
  126. // TVmaze's lookup by one external id: { id } (null = TVmaze doesn't know it, 404) or { failed }. hl:fetch follows the
  127. // 301 to /shows/<id> by itself, so a hit answers the show's own JSON.
  128. static tvmazeLookup = (param, value) => {
  129. let r = fetch(tvmazeBase + '/lookup/shows?' + param + '=' + value, { headers = { 'accept' = 'application/json' 'user-agent' = 'tracker.worldapi.org (tmdb sync)' } timeoutMs = 20000 })
  130. if (r == null || r.status == null || r.status == 0) { return { failed = 'TVmaze: no answer for ' + param + '=' + value } }
  131. if (r.status == 404) { return { id = null } }
  132. if (r.status != 200) { return { failed = 'TVmaze: HTTP ' + r.status + ' for ' + param + '=' + value } }
  133. let j = r.json()
  134. if (j == null || numIdOr(j.id) == null) { return { failed = 'TVmaze: no id for ' + param + '=' + value } }
  135. return { id = j.id }
  136. }
  137. // the ids to SET on a show from TMDB's external_ids (+ TVmaze for a series): { imdbId, tvdbId, tvmzId } — each the
  138. // show's own value when it has one (never overwritten), else the found one, else null — plus { filled, tvmazeRequests,
  139. // failed }. TVmaze is asked by imdb, then by thetvdb, only while the show (after TMDB) still has no tvmzId — and only
  140. // when `askTvmaze` (the adult backfill does not: one request per title)
  141. static externalIdsFor = (show, ext, isSeries, askTvmaze) => {
  142. let out = { imdbId = show.imdbId tvdbId = show.tvdbId tvmzId = show.tvmzId filled = 0 tvmazeRequests = 0 failed = null }
  143. let e = ext != null ? ext : {}
  144. if (!hasId(out.imdbId) && imdbOr(e.imdb_id) != null) { out.imdbId = e.imdb_id out.filled = out.filled + 1 }
  145. if (!isSeries) { return out }
  146. if (!hasId(out.tvdbId) && numIdOr(e.tvdb_id) != null) { out.tvdbId = e.tvdb_id out.filled = out.filled + 1 }
  147. if (hasId(out.tvmzId) || !askTvmaze) { return out }
  148. let tries = []
  149. if (imdbOr('' + out.imdbId) != null) { tries.push({ param = 'imdb' value = '' + out.imdbId }) }
  150. if (numIdOr(out.tvdbId) != null) { tries.push({ param = 'thetvdb' value = '' + out.tvdbId }) }
  151. for (t of tries) {
  152. if (!hasId(out.tvmzId) && out.failed == null) {
  153. out.tvmazeRequests = out.tvmazeRequests + 1
  154. let found = tvmazeLookup(t.param, t.value)
  155. if (found.failed != null) { out.failed = found.failed } else if (found.id != null) { out.tvmzId = found.id out.filled = out.filled + 1 }
  156. }
  157. }
  158. return out
  159. }
  160. // ---- TVmaze merge (tracker.worldapi.org#15) ------------------------------------------------------
  161. // a title that is none yet: empty, "TBA"/"TBD", or "Episode <n>" (what TMDB and TVmaze name an untitled episode; also
  162. // French/German/Spanish: real data has TVmaze "Épisode 1" where TMDB says "Episode 1")
  163. static placeholderWords = ['episode', 'épisode', 'Épisode', 'folge', 'episodio']
  164. static isPlaceholderTitle = (v) => {
  165. let t = textOr(v)
  166. if (t == null) { return true }
  167. let l = t.trim().toLowerCase()
  168. if (l == 'tba' || l == 'tbd' || l == 'tba.' || l == 'to be announced') { return true }
  169. let parts = l.split(' ')
  170. if (parts.length != 2 || !placeholderWords.includes(parts[0]) || parts[1].length == 0) { return false }
  171. let i = 0
  172. while (i < parts[1].length) {
  173. if (!'0123456789'.includes(parts[1][i])) { return false }
  174. i = i + 1
  175. }
  176. return true
  177. }
  178. // 'YYYY-MM-DD' → a day number (only differences matter), null when it is no such date
  179. static dayNumber = (v) => {
  180. let t = textOr(v)
  181. if (t == null || t.length < 10) { return null }
  182. let y = toNumber(t.slice(0, 4))
  183. let m = toNumber(t.slice(5, 7))
  184. let d = toNumber(t.slice(8, 10))
  185. if (y == null || m == null || d == null || m < 1 || m > 12) { return null }
  186. let before = [0, 31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334]
  187. let py = m <= 2 ? y - 1 : y
  188. let leaps = (py - py % 4) / 4 - (py - py % 100) / 100 + (py - py % 400) / 400
  189. return y * 365 + leaps + before[m - 1] + d
  190. }
  191. static daysApart = (a, b) => {
  192. let x = dayNumber(a)
  193. let y = dayNumber(b)
  194. if (x == null || y == null) { return null }
  195. return x > y ? x - y : y - x
  196. }
  197. // TVmaze's text without its HTML (tracker#18: `tvmazeSummary`): tags dropped — a block tag (p, br, div, li) becomes a space —
  198. // the common entities decoded, white space collapsed; null when nothing is left
  199. static blockTags = ['p', 'br', 'div', 'li', 'ul', 'ol', 'h1', 'h2', 'h3', 'h4', 'blockquote']
  200. static entities = [['&nbsp;', ' '], ['&quot;', '"'], ['&#39;', "'"], ['&apos;', "'"], ['&lt;', '<'], ['&gt;', '>'], ['&ndash;', '–'], ['&mdash;', '—'], ['&hellip;', '…'], ['&amp;', '&']]
  201. static stripHtml = (v) => {
  202. let t = textOr(v)
  203. if (t == null) { return null }
  204. let out = ''
  205. let i = 0
  206. while (i < t.length) {
  207. if (t[i] == '<') {
  208. let close = t.indexOf('>', i)
  209. if (close < 0) { close = t.length }
  210. let name = t.slice(i + 1, close).trim().toLowerCase()
  211. if (name.startsWith('/')) { name = name.slice(1) }
  212. let sp = name.indexOf(' ')
  213. if (sp >= 0) { name = name.slice(0, sp) }
  214. if (name.endsWith('/')) { name = name.slice(0, name.length - 1) }
  215. if (blockTags.includes(name)) { out = out + ' ' }
  216. i = close + 1
  217. } else {
  218. // a line break / tab (a control character) is a space too (strings have no escapes to name them)
  219. out = out + (t.charCodeAt(i) < 32 ? ' ' : t[i])
  220. i = i + 1
  221. }
  222. }
  223. for (e of entities) { out = out.replaceAll(e[0], e[1]) }
  224. let words = []
  225. for (w of out.split(' ')) { if (w != '') { words.push(w) } }
  226. return words.length > 0 ? words.join(' ') : null
  227. }
  228. // TVmaze's show WITH its episode list, one request (tracker#18: `?embed=episodes` brings the show's own summary too — before
  229. // it was `/shows/<id>/episodes`): { list, summary (HTML stripped, or null) } or { failed }
  230. static tvmazeEpisodes = (tvmzId) => {
  231. let r = fetch(tvmazeBase + '/shows/' + tvmzId + '?embed=episodes', { headers = { 'accept' = 'application/json' 'user-agent' = 'tracker.worldapi.org (tmdb sync)' } timeoutMs = 20000 })
  232. if (r == null || r.status == null || r.status == 0) { return { failed = 'TVmaze: no answer for episodes of ' + tvmzId } }
  233. if (r.status != 200) { return { failed = 'TVmaze: HTTP ' + r.status + ' for episodes of ' + tvmzId } }
  234. let j = r.json()
  235. if (j == null || j._embedded == null || j._embedded.episodes == null || j._embedded.episodes.length == null) { return { failed = 'TVmaze: no episode list for ' + tvmzId } }
  236. return { list = j._embedded.episodes summary = stripHtml(j.summary) }
  237. }
  238. // do we and TVmaze number a show the same way? Of the episodes both know (same S/E), at least 80 % must air within a day
  239. // of each other OR carry the same title (a streaming show TVmaze dates all on its release day, TMDB week by week, still
  240. // agrees by title). A show we have episodes of but TVmaze numbers none of the same (e.g. seasons by year) never agrees.
  241. // Answers { known, compared, agree, ok }.
  242. static sameTitle = (a, b) => { return !isPlaceholderTitle(a) && !isPlaceholderTitle(b) && a.trim().toLowerCase() == b.trim().toLowerCase() }
  243. static numberingAgrees = (epByNumber, list) => {
  244. let out = { known = 0 compared = 0 agree = 0 ok = true }
  245. let localCount = epByNumber.keys().length
  246. for (te of list) {
  247. let e = epByNumber[te.season + 'x' + te.number]
  248. if (e != null) {
  249. out.known = out.known + 1
  250. let apart = daysApart(e.release, te.airdate)
  251. let titled = !isPlaceholderTitle(e.title) && !isPlaceholderTitle(te.name)
  252. if (apart != null || titled) {
  253. out.compared = out.compared + 1
  254. if ((apart != null && apart <= 1) || sameTitle(e.title, te.name)) { out.agree = out.agree + 1 }
  255. }
  256. }
  257. }
  258. out.ok = !((localCount > 0 && out.known == 0) || (out.compared > 0 && out.agree * 5 < out.compared * 4))
  259. return out
  260. }
  261. // Merge TVmaze's episodes of a series into what we have AFTER the TMDB step (gaps only, rules at the top).
  262. // `seasonsByNumber` / `epByNumber` = the show's seasons by number and episodes by '<s>x<e>', as the TMDB step left them.
  263. // Answers { tvmazeRequests, newSeasons, newEpisodes, filledEpisodes, skipped, errors, error, showSeasons }.
  264. static mergeTvmaze = (title, showKey, tvmzId, seasonsIn, episodesIn, showSeasonsIn) => {
  265. let seasonsByNumber = seasonsIn
  266. let epByNumber = episodesIn
  267. let out = { tvmazeRequests = 1 newSeasons = 0 newEpisodes = 0 filledEpisodes = 0 skipped = 0 errors = 0 error = '' showSeasons = showSeasonsIn summary = null }
  268. let got = tvmazeEpisodes(tvmzId)
  269. if (got.failed != null) { out.errors = 1 out.error = title + ': ' + got.failed return out }
  270. out.summary = got.summary
  271. // the numbered episodes TVmaze has in seasons we take (no specials; season 0 only when the show has one)
  272. let list = []
  273. for (te of got.list) {
  274. if (numIdOr(te.season) != null && numIdOr(te.number) != null) { list.push(te) }
  275. else if (te.season == 0 && numIdOr(te.number) != null && seasonsByNumber['0'] != null) { list.push(te) }
  276. }
  277. if (!numberingAgrees(epByNumber, list).ok) { out.skipped = 1 return out }
  278. let added = {}
  279. let addedFirstDate = {}
  280. for (te of list) {
  281. let key = te.season + 'x' + te.number
  282. let e = epByNumber[key]
  283. if (e == null) {
  284. let seasonKey = seasonsByNumber['' + te.season] != null ? seasonsByNumber['' + te.season].id : null
  285. if (seasonKey == null) {
  286. if (added['' + te.season] == null) { added['' + te.season] = [] addedFirstDate['' + te.season] = textOr(te.airdate) seasonsByNumber['' + te.season] = { id = newId() isNew = true } }
  287. seasonKey = seasonsByNumber['' + te.season].id
  288. }
  289. let fresh = { id = newId() episodeNumber = te.number seasonNumber = te.season
  290. title = textOr(te.name) summary = '' release = textOr(te.airdate)
  291. show = showKey season = seasonKey tmdbId = null tvmzId = te.id oldId = 'tvmaze-episode:' + te.id }
  292. if (episodesTable.put(fresh) == null) {
  293. out.errors = out.errors + 1
  294. out.error = title + ': TVmaze episode ' + te.id + ' not stored: ' + episodesTable.lastError()
  295. } else {
  296. if (added['' + te.season] == null) { added['' + te.season] = [] }
  297. let l = added['' + te.season]
  298. l.push(fresh.id)
  299. added['' + te.season] = l
  300. epByNumber[key] = fresh
  301. out.newEpisodes = out.newEpisodes + 1
  302. }
  303. } else {
  304. let changed = false
  305. if (isPlaceholderTitle(e.title) && !isPlaceholderTitle(te.name)) { e.title = te.name changed = true }
  306. if (textOr(e.release) == null && textOr(te.airdate) != null) { e.release = te.airdate changed = true }
  307. if (changed) {
  308. if (episodesTable.update(e.id, e) == null) {
  309. out.errors = out.errors + 1
  310. out.error = title + ': episode ' + e.id + ' not filled from TVmaze: ' + episodesTable.lastError()
  311. } else {
  312. epByNumber[key] = e
  313. out.filledEpisodes = out.filledEpisodes + 1
  314. }
  315. }
  316. }
  317. }
  318. // the seasons that got episodes: an existing one is updated, a new one stored and linked to the show
  319. let showSeasons = []
  320. for (sid of showSeasonsIn) { showSeasons.push(sid) }
  321. for (n of added.keys()) {
  322. let ids = added[n]
  323. let s = seasonsByNumber[n]
  324. if (ids.length > 0 && s.isNew == true) {
  325. let season = { id = s.id seasonNumber = toNumber(n) title = 'Season ' + n release = addedFirstDate[n] summary = ''
  326. show = showKey episodes = ids oldId = 'tvmaze-season:' + tvmzId + ':' + n }
  327. if (seasonsTable.put(season) == null) {
  328. out.errors = out.errors + 1
  329. out.error = title + ': TVmaze season ' + n + ' not stored: ' + seasonsTable.lastError()
  330. } else {
  331. showSeasons.push(season.id)
  332. out.newSeasons = out.newSeasons + 1
  333. }
  334. } else if (ids.length > 0) {
  335. let season = seasonsTable.fetch(s.id)
  336. let eps = []
  337. for (eid of season.episodes) { eps.push(eid) }
  338. for (eid of ids) { eps.push(eid) }
  339. season.episodes = eps
  340. if (seasonsTable.update(season.id, season) == null) {
  341. out.errors = out.errors + 1
  342. out.error = title + ': season ' + season.id + ' not updated: ' + seasonsTable.lastError()
  343. }
  344. }
  345. }
  346. out.showSeasons = showSeasons
  347. return out
  348. }
  349. // ---- posters ----------------------------------------------------------------------------------
  350. // the extension of a TMDB poster_path ('/abc.jpg' → 'jpg'); only the ones the poster route knows
  351. static posterExt = (posterPath) => {
  352. let parts = posterPath.split('.')
  353. let ext = parts[parts.length - 1].toLowerCase()
  354. if (ext == 'jpg' || ext == 'jpeg' || ext == 'png' || ext == 'webp') { return ext }
  355. return null
  356. }
  357. // downloads the w342 poster when the file is missing or TMDB has a new one; answers { requests, saved, failed }
  358. // and sets show.image / show.tmdbPoster on the (caller's) record — the file is <oldId>.<ext>, the name
  359. // shows.hl posterName() already builds, served by project.hl's /posters/:name route
  360. static syncPoster = (show, posterPath) => {
  361. let out = { requests = 0 saved = 0 failed = null image = show.image tmdbPoster = show.tmdbPoster }
  362. if (textOr(posterPath) == null || show.oldId == null) { return out }
  363. let ext = posterExt(posterPath)
  364. if (ext == null) { return out }
  365. let file = postersDir + '/' + show.oldId + '.' + ext
  366. if (exists(file) && show.tmdbPoster == posterPath && show.image == ext) { return out }
  367. out.requests = 1
  368. let r = fetch(tmdbImageBase + '/w342' + posterPath, { headers = { 'user-agent' = 'tracker.worldapi.org (tmdb sync)' } timeoutMs = 20000 })
  369. if (r == null || r.status != 200 || r.body == null || r.body.length == 0) {
  370. out.failed = 'poster ' + posterPath + ': ' + (r == null ? 'no answer' : 'HTTP ' + r.status)
  371. return out
  372. }
  373. mkDir(postersDir)
  374. writeFile(file, toBytes(r.body), 420)
  375. out.saved = 1
  376. out.image = ext
  377. out.tmdbPoster = posterPath
  378. return out
  379. }
  380. // ---- one show -----------------------------------------------------------------------------------
  381. static newId = () => { return randomBytes(8, 'hex') }
  382. // TMDB's adult flag of a details/search answer: true / false, null when it has none (mission 054)
  383. static adultOf = (j) => {
  384. if (j == null) { return null }
  385. if (j.adult == true) { return true }
  386. if (j.adult == false) { return false }
  387. return null
  388. }
  389. // which TMDB seasons to sync: every numbered one with episodes; 0 only when the show has a season 0 already
  390. static wantedSeasonNumbers = (details, localByNumber) => {
  391. let out = []
  392. if (details.seasons == null) { return out }
  393. for (s of details.seasons) {
  394. let n = s.season_number
  395. if (n != null && (n > 0 || localByNumber['' + n] != null) && (s.episode_count == null || s.episode_count > 0 || localByNumber['' + n] != null)) { out.push(n) }
  396. }
  397. return out
  398. }
  399. // the seasons in batches of 20 (TMDB's append_to_response limit)
  400. static batchesOf = (list, size) => {
  401. let out = []
  402. let cur = []
  403. for (x of list) {
  404. cur.push(x)
  405. if (cur.length == size) { out.push(cur) cur = [] }
  406. }
  407. if (cur.length > 0) { out.push(cur) }
  408. return out
  409. }
  410. // fields of an episode/season from TMDB that differ from the stored record (null/empty never overwrites)
  411. static changedText = (stored, fresh) => { return fresh != null && fresh != stored }
  412. // mission 070 (conductor: no step destroys migrated data): a season / episode the old tracker's migration brought (its `oldId`
  413. // is neither `tmdb-…` nor `tvmaze-…`) keeps its FIRST title / summary in `migratedTitle` / `migratedSummary` before TMDB's text
  414. // replaces it (set once; shown nowhere) — the daily delta run now also refreshes seasons of titles nobody follows
  415. static isMigratedRow = (r) => { return r.oldId != null && ('' + r.oldId).slice(0, 5) != 'tmdb-' && ('' + r.oldId).slice(0, 7) != 'tvmaze-' }
  416. // a MOVIE: one request — its imdb id (if missing) and its poster; nothing else of the record is touched. `pre` = that
  417. // request's answer when the caller already made it (details.hl firstStep, tracker#26), else null
  418. static syncMovie = (show, res, pre) => {
  419. let out = res
  420. out.requests = 1
  421. let details = pre != null ? pre : tmdbGet('/movie/' + show.tmdbId + '?append_to_response=external_ids,credits')
  422. if (details.failed != null) { out.errors = 1 out.error = show.title + ': ' + details.failed return out }
  423. if (pre == null) { out.details = details }
  424. let ids = externalIdsFor(show, details.external_ids, false, false)
  425. out.ids = ids.filled
  426. let poster = syncPoster(show, details.poster_path)
  427. out.requests = out.requests + poster.requests
  428. out.posters = poster.saved
  429. if (poster.failed != null) { out.errors = out.errors + 1 out.error = show.title + ': ' + poster.failed }
  430. let m = show
  431. m.imdbId = ids.imdbId
  432. m.image = poster.image
  433. m.tmdbPoster = poster.tmdbPoster
  434. if (adultOf(details) != null) { m.adult = adultOf(details) }
  435. m.tmdbSync = now()
  436. if (showsTable.update(m.id, m) == null) { out.errors = out.errors + 1 out.error = show.title + ': movie not updated: ' + showsTable.lastError() }
  437. return out
  438. }
  439. // Sync ONE show (by our id). Answers counts: { requests, tvmazeRequests, newSeasons, newEpisodes, updatedEpisodes,
  440. // updatedSeasons, posters, ids, tvmazeEpisodes, tvmazeSeasons, tvmazeFilled, tvmazeSkipped, errors, error } + `details` (the
  441. // details answer with its credits when this call asked for it, else null — tracker#28: the caller stores the credits). Never throws for a TMDB failure — it is counted and the show left as is.
  442. static syncShow = (showId) => { return syncShowPart(showId, null, null) }
  443. // the same with the details request (`/tv|movie/<id>?append_to_response=external_ids…`) already answered — `pre`, counted as
  444. // this show's first request — so details.hl (tracker#26) asks TMDB once for details, ids AND the cast
  445. static syncShowWith = (showId, pre) => { return syncShowPart(showId, pre, null) }
  446. // ONE PART of a series' sync (tracker#26: a talk show with 4,000 episodes took 10 s in one step, blocking every request):
  447. // `only` = { seasons (the TMDB season numbers of this part), finish (false: only those seasons — new ones linked to the show,
  448. // nothing else; true: the ids, TVmaze, poster and the show record), tvmaze (false: no TVmaze lookup/merge — tracker#18's light
  449. // step) }; null = everything at once. `pre` is required with `only` (its request is the caller's: a part counts only its own
  450. // requests).
  451. static emptyResult = () => { return { requests = 0 tvmazeRequests = 0 newSeasons = 0 newEpisodes = 0 updatedEpisodes = 0 updatedSeasons = 0 posters = 0 ids = 0 tvmazeEpisodes = 0 tvmazeSeasons = 0 tvmazeFilled = 0 tvmazeSkipped = 0 errors = 0 error = '' details = null } }
  452. static syncShowPart = (showId, pre, only) => {
  453. let res = emptyResult()
  454. let show = showsTable.fetch(showId)
  455. if (show == null) { res.errors = 1 res.error = 'no show ' + showId return res }
  456. if (show.tmdbId == null) { res.errors = 1 res.error = show.title + ': no tmdbId' return res }
  457. if (show.type == 'movie') { return syncMovie(show, res, pre) }
  458. res.requests = only != null ? 0 : 1
  459. let details = pre != null ? pre : tmdbGet('/tv/' + show.tmdbId + '?append_to_response=external_ids,aggregate_credits')
  460. if (details.failed != null) { res.errors = 1 res.error = show.title + ': ' + details.failed return res }
  461. if (pre == null) { res.details = details }
  462. // what we have: seasons by number, episodes by tmdbId and by S/E number
  463. // (tracker#18: a part reads only ITS seasons' episodes, unless it finishes with the TVmaze merge, which compares them all —
  464. // Today, 15,716 episodes, read them all for a one-season change: 20 s and GBs of memory)
  465. let tvmazeToo = only == null || only.tvmaze != false
  466. let allEpisodes = only == null || (only.finish && tvmazeToo)
  467. let seasonsByNumber = {}
  468. let epByTmdb = {}
  469. let epByNumber = {}
  470. for (sid of show.seasons) {
  471. let s = seasonsTable.fetch(sid)
  472. if (s != null) {
  473. seasonsByNumber['' + s.seasonNumber] = s
  474. let epIds = allEpisodes || only.seasons.includes(s.seasonNumber) ? s.episodes : []
  475. for (eid of epIds) {
  476. let e = episodesTable.fetch(eid)
  477. if (e != null) {
  478. if (e.tmdbId != null) { epByTmdb['' + e.tmdbId] = e }
  479. epByNumber[e.seasonNumber + 'x' + e.episodeNumber] = e
  480. }
  481. }
  482. }
  483. }
  484. // plain locals for the record literals below: inside a literal, an entry named `show`/`season` would shadow
  485. // the variable for every later entry (SPEC "A literal's entries are evaluated in order")
  486. let showKey = show.id
  487. let showTmdb = show.tmdbId
  488. let showSeasons = []
  489. for (sid of show.seasons) { showSeasons.push(sid) }
  490. let showChanged = false
  491. let wanted = wantedSeasonNumbers(details, seasonsByNumber)
  492. if (only != null) {
  493. let part = []
  494. for (n of wanted) { if (only.seasons.includes(n)) { part.push(n) } }
  495. wanted = part
  496. }
  497. for (batch of batchesOf(wanted, 20)) {
  498. let names = []
  499. for (n of batch) { names.push('season/' + n) }
  500. res.requests = res.requests + 1
  501. let withSeasons = tmdbGet('/tv/' + show.tmdbId + '?append_to_response=' + names.join(','))
  502. if (withSeasons.failed != null) {
  503. res.errors = res.errors + 1
  504. res.error = show.title + ': ' + withSeasons.failed
  505. } else {
  506. for (n of batch) {
  507. let ts = withSeasons['season/' + n]
  508. if (ts != null && ts.episodes != null) {
  509. let season = seasonsByNumber['' + n]
  510. let seasonIsNew = season == null
  511. let seasonChanged = false
  512. if (seasonIsNew) {
  513. season = { id = newId() seasonNumber = n title = textOr(ts.name) != null ? ts.name : 'Season ' + n
  514. release = textOr(ts.air_date) summary = textOr(ts.overview) != null ? ts.overview : ''
  515. show = showKey episodes = [] oldId = 'tmdb-season:' + showTmdb + ':' + n }
  516. } else {
  517. let seasonMigrated = isMigratedRow(season)
  518. if (changedText(season.title, textOr(ts.name))) {
  519. if (seasonMigrated && season.migratedTitle == null && textOr(season.title) != null) { season.migratedTitle = season.title }
  520. season.title = ts.name seasonChanged = true
  521. }
  522. if (changedText(season.summary, textOr(ts.overview))) {
  523. if (seasonMigrated && season.migratedSummary == null && textOr(season.summary) != null) { season.migratedSummary = season.summary }
  524. season.summary = ts.overview seasonChanged = true
  525. }
  526. if (changedText(season.release, textOr(ts.air_date))) { season.release = ts.air_date seasonChanged = true }
  527. }
  528. let seasonKey = season.id
  529. let seasonEpisodes = []
  530. for (eid of season.episodes) { seasonEpisodes.push(eid) }
  531. for (te of ts.episodes) {
  532. if (te.id != null && te.episode_number != null) {
  533. let e = epByTmdb['' + te.id]
  534. if (e == null) {
  535. let byNum = epByNumber[n + 'x' + te.episode_number]
  536. if (byNum != null && byNum.tmdbId == null) { e = byNum }
  537. }
  538. if (e == null) {
  539. let fresh = { id = newId() episodeNumber = te.episode_number seasonNumber = n
  540. title = textOr(te.name) summary = textOr(te.overview) != null ? te.overview : '' release = textOr(te.air_date)
  541. show = showKey season = seasonKey tmdbId = te.id oldId = 'tmdb-episode:' + te.id }
  542. let putId = episodesTable.put(fresh)
  543. if (putId == null) {
  544. res.errors = res.errors + 1
  545. res.error = show.title + ': episode ' + te.id + ' not stored: ' + episodesTable.lastError()
  546. } else {
  547. seasonEpisodes.push(fresh.id)
  548. epByTmdb['' + te.id] = fresh
  549. epByNumber[n + 'x' + te.episode_number] = fresh
  550. res.newEpisodes = res.newEpisodes + 1
  551. seasonChanged = true
  552. }
  553. } else {
  554. let changed = false
  555. if (e.tmdbId == null) { e.tmdbId = te.id changed = true }
  556. // #15: TMDB's "Episode 4" never replaces a real title (one TVmaze filled, or the migrated one)
  557. let epMigrated = isMigratedRow(e)
  558. if (changedText(e.title, textOr(te.name)) && !(isPlaceholderTitle(te.name) && !isPlaceholderTitle(e.title))) {
  559. if (epMigrated && e.migratedTitle == null && textOr(e.title) != null) { e.migratedTitle = e.title }
  560. e.title = te.name changed = true
  561. }
  562. if (changedText(e.summary, textOr(te.overview))) {
  563. if (epMigrated && e.migratedSummary == null && textOr(e.summary) != null) { e.migratedSummary = e.summary }
  564. e.summary = te.overview changed = true
  565. }
  566. if (changedText(e.release, textOr(te.air_date))) { e.release = te.air_date changed = true }
  567. if (changed) {
  568. if (episodesTable.update(e.id, e) == null) {
  569. res.errors = res.errors + 1
  570. res.error = show.title + ': episode ' + e.id + ' not updated: ' + episodesTable.lastError()
  571. } else {
  572. epByTmdb['' + e.tmdbId] = e
  573. epByNumber[e.seasonNumber + 'x' + e.episodeNumber] = e
  574. res.updatedEpisodes = res.updatedEpisodes + 1
  575. }
  576. }
  577. }
  578. }
  579. }
  580. season.episodes = seasonEpisodes
  581. if (seasonIsNew) {
  582. if (seasonsTable.put(season) == null) {
  583. res.errors = res.errors + 1
  584. res.error = show.title + ': season ' + n + ' not stored: ' + seasonsTable.lastError()
  585. } else {
  586. seasonsByNumber['' + n] = season
  587. showSeasons.push(season.id)
  588. showChanged = true
  589. res.newSeasons = res.newSeasons + 1
  590. }
  591. } else if (seasonChanged) {
  592. if (seasonsTable.update(season.id, season) == null) {
  593. res.errors = res.errors + 1
  594. res.error = show.title + ': season ' + season.id + ' not updated: ' + seasonsTable.lastError()
  595. } else {
  596. res.updatedSeasons = res.updatedSeasons + 1
  597. }
  598. }
  599. }
  600. }
  601. }
  602. }
  603. // a part that is not the last: only the new seasons are linked to the show (the next part reads them)
  604. if (only != null && !only.finish) {
  605. if (showChanged) {
  606. show.seasons = showSeasons
  607. if (showsTable.update(show.id, show) == null) { res.errors = res.errors + 1 res.error = show.title + ': show not updated: ' + showsTable.lastError() }
  608. }
  609. return res
  610. }
  611. // the external ids (tracker.worldapi.org#12): missing ones from TMDB's external_ids, a missing TVmaze id from TVmaze
  612. let ids = externalIdsFor(show, details.external_ids, true, tvmazeToo)
  613. res.ids = ids.filled
  614. res.tvmazeRequests = ids.tvmazeRequests
  615. if (ids.failed != null) { res.errors = res.errors + 1 res.error = show.title + ': ' + ids.failed }
  616. // tracker.worldapi.org#15: what TVmaze has and TMDB not yet (new episodes/seasons, empty titles and air dates)
  617. let tvmazeSummary = null
  618. if (numIdOr(ids.tvmzId) != null && tvmazeToo) {
  619. let tvm = mergeTvmaze(show.title, showKey, ids.tvmzId, seasonsByNumber, epByNumber, showSeasons)
  620. res.tvmazeRequests = res.tvmazeRequests + tvm.tvmazeRequests
  621. res.tvmazeEpisodes = tvm.newEpisodes
  622. res.tvmazeSeasons = tvm.newSeasons
  623. res.tvmazeFilled = tvm.filledEpisodes
  624. res.tvmazeSkipped = tvm.skipped
  625. if (tvm.errors > 0) { res.errors = res.errors + tvm.errors res.error = tvm.error }
  626. showSeasons = tvm.showSeasons
  627. tvmazeSummary = tvm.summary
  628. }
  629. // the show itself: status, counts, TMDB's overview, the poster, when it was synced
  630. let poster = syncPoster(show, details.poster_path)
  631. res.requests = res.requests + poster.requests
  632. res.posters = poster.saved
  633. if (poster.failed != null) { res.errors = res.errors + 1 res.error = show.title + ': ' + poster.failed }
  634. show.seasons = showSeasons
  635. if (textOr(details.status) != null) { show.status = details.status }
  636. if (details.number_of_seasons != null) { show.seasonsCount = details.number_of_seasons }
  637. if (details.number_of_episodes != null) { show.episodesCount = details.number_of_episodes }
  638. // tracker#18: TMDB's text → tmdbSummary, TVmaze's → tvmazeSummary; `summary` is the creator's own text, never written here
  639. if (textOr(details.overview) != null) { show.tmdbSummary = details.overview }
  640. if (tvmazeSummary != null) { show.tvmazeSummary = tvmazeSummary }
  641. show.image = poster.image
  642. show.tmdbPoster = poster.tmdbPoster
  643. show.imdbId = ids.imdbId
  644. show.tvdbId = ids.tvdbId
  645. show.tvmzId = ids.tvmzId
  646. if (adultOf(details) != null) { show.adult = adultOf(details) }
  647. if (textOr(details.type) != null) { show.tmdbType = details.type show.kind = kindOfTitle(details.type, show.genres) }
  648. show.tmdbSync = now()
  649. if (showsTable.update(show.id, show) == null) { res.errors = res.errors + 1 res.error = show.title + ': show not updated: ' + showsTable.lastError() }
  650. return res
  651. }
  652. // ---- the LIGHT step of the daily delta run (tracker#18) ---------------------------------------------------------------
  653. // A changed title NOBODY follows (deltasync.hl): its details (+ ids + credits: the record, applySynced) and only the seasons
  654. // that can have changed — the ones we don't have, the ones where TMDB counts MORE episodes than we have (fewer: TVmaze's
  655. // extras, nothing to fetch), and the newest one (new or renamed episodes land there). No TVmaze (its merge only matters for the followed shows' lists; their full step still does
  656. // it). Real data (2026-10-01): Today (67 seasons / 15,716 episodes) and EastEnders (43 / 7,349) change every day; their full
  657. // sync blocked the server 20.8 s each and grew the process by ~5 GB each.
  658. static changedSeasonNumbers = (details, show) => {
  659. let ours = {}
  660. for (sid of show.seasons) {
  661. let s = seasonsTable.fetch(sid)
  662. if (s != null) { ours['' + s.seasonNumber] = s.episodes.length }
  663. }
  664. let latest = -1
  665. for (x of (details.seasons != null ? details.seasons : [])) {
  666. if (x != null && x.season_number != null && x.season_number > latest && (x.episode_count == null || x.episode_count > 0)) { latest = x.season_number }
  667. }
  668. let out = []
  669. for (x of (details.seasons != null ? details.seasons : [])) {
  670. if (x != null && x.season_number != null) {
  671. let n = x.season_number
  672. if (n == latest || ours['' + n] == null || (x.episode_count != null && x.episode_count > ours['' + n])) { out.push(n) }
  673. }
  674. }
  675. return out
  676. }
  677. static syncShowDelta = (showId) => {
  678. let show = showsTable.fetch(showId)
  679. if (show == null || show.tmdbId == null || show.type == 'movie') { return syncShow(showId) }
  680. let details = tmdbGet('/tv/' + show.tmdbId + '?append_to_response=external_ids,aggregate_credits')
  681. if (details.failed != null) {
  682. let res = emptyResult()
  683. res.requests = 1
  684. res.errors = 1
  685. res.error = show.title + ': ' + details.failed
  686. return res
  687. }
  688. let r = syncShowPart(showId, details, { seasons = changedSeasonNumbers(details, show) finish = true tvmaze = false })
  689. r.requests = r.requests + 1
  690. r.details = details
  691. return r
  692. }
  693. // ---- the adult backfill (mission 054) ------------------------------------------------------------
  694. // a title still to ask: its adult flag unknown, it has a tmdbId and TMDB never answered "no such title" for it
  695. static needsAdultFlag = (show) => { return show.adult != true && show.adult != false && show.adultCheck == null && show.tmdbId != null }
  696. // the ids of every title the backfill still has to ask, in table order (read once when the backfill starts — so after a
  697. // restart it goes on with what is left)
  698. static adultBackfillQueue = () => {
  699. let out = []
  700. for (s of showsTable.find(null, null)) { if (needsAdultFlag(s)) { out.push(s.id) } }
  701. return out
  702. }
  703. // ONE title: one TMDB details request (adult, poster_path, external ids) + its poster when the file is missing. Answers
  704. // { requests (TMDB API requests: what the pause counts), posters, adult (true / false / null), ids, errors, error, retry
  705. // (a passing failure: no answer, 429, 5xx — ask again later) }. A 404 marks the title `adultCheck` (never asked again).
  706. static backfillTitle = (showId) => {
  707. let out = { requests = 0 posters = 0 adult = null ids = 0 errors = 0 error = '' retry = false }
  708. let show = showsTable.fetch(showId)
  709. if (show == null || !needsAdultFlag(show)) { return out }
  710. let movie = show.type == 'movie'
  711. out.requests = 1
  712. let details = tmdbGet((movie ? '/movie/' : '/tv/') + show.tmdbId + '?append_to_response=external_ids')
  713. if (details.failed != null) {
  714. out.errors = 1
  715. out.error = show.title + ': ' + details.failed
  716. if (details.status == 404) {
  717. show.adultCheck = 'TMDB 404'
  718. showsTable.update(show.id, show)
  719. } else { out.retry = true }
  720. return out
  721. }
  722. let ids = externalIdsFor(show, details.external_ids, !movie, false)
  723. out.ids = ids.filled
  724. out.adult = adultOf(details)
  725. // an ADULT title's poster is never shown publicly → not downloaded (mission 056: ~170 MB of the live data); its flag and
  726. // ids are stored as for any other title
  727. let poster = { saved = 0 failed = null image = show.image tmdbPoster = show.tmdbPoster }
  728. if (out.adult != true) { poster = syncPoster(show, details.poster_path) }
  729. out.posters = poster.saved
  730. if (poster.failed != null) { out.errors = out.errors + 1 out.error = show.title + ': ' + poster.failed }
  731. if (out.adult == null) { show.adultCheck = 'TMDB gave no adult flag' } else { show.adult = out.adult }
  732. if (!movie && textOr(details.type) != null) { show.tmdbType = details.type show.kind = kindOfTitle(details.type, show.genres) }
  733. show.imdbId = ids.imdbId
  734. show.tvdbId = ids.tvdbId
  735. show.image = poster.image
  736. show.tmdbPoster = poster.tmdbPoster
  737. if (showsTable.update(show.id, show) == null) { out.errors = out.errors + 1 out.error = show.title + ': not updated: ' + showsTable.lastError() }
  738. return out
  739. }
  740. // ---- the kind backfill (tracker.worldapi.org#21, mission 068) ----------------------------------------------------
  741. // a TV title whose kind is not stored yet
  742. static needsKind = (show) => { return show.type == 'series' && show.kind != 'series' && show.kind != 'show' }
  743. // the ids of every TV title still without a kind, in table order (read once when the backfill starts — it resumes)
  744. static kindBackfillQueue = () => {
  745. let out = []
  746. for (s of showsTable.find(null, null)) { if (needsKind(s)) { out.push(s.id) } }
  747. return out
  748. }
  749. // ONE title → { requests (TMDB API requests: what the pause counts), kind ('series' | 'show' | null), fromGenre, errors, error,
  750. // retry (a passing failure: no answer, 429, 5xx) }
  751. static kindTitle = (showId) => {
  752. let out = { requests = 0 kind = null fromGenre = false errors = 0 error = '' retry = false }
  753. let show = showsTable.fetch(showId)
  754. if (show == null || !needsKind(show)) { return out }
  755. if (hasShowGenre(show.genres) || show.tmdbId == null) {
  756. out.fromGenre = true
  757. show.kind = kindOfTitle(null, show.genres)
  758. if (!hasShowGenre(show.genres)) { show.kindCheck = 'no tmdbId' }
  759. } else {
  760. out.requests = 1
  761. let details = tmdbGet('/tv/' + show.tmdbId)
  762. if (details.failed != null) {
  763. out.errors = 1
  764. out.error = show.title + ': ' + details.failed
  765. if (details.status != 404) { out.retry = true return out }
  766. show.kindCheck = 'TMDB 404'
  767. show.kind = kindOfTitle(null, show.genres)
  768. } else {
  769. if (textOr(details.type) != null) { show.tmdbType = details.type } else { show.kindCheck = 'TMDB gave no type' }
  770. show.kind = kindOfTitle(textOr(details.type), show.genres)
  771. }
  772. }
  773. out.kind = show.kind
  774. if (showsTable.update(show.id, show) == null) { out.errors = out.errors + 1 out.error = show.title + ': not updated: ' + showsTable.lastError() }
  775. return out
  776. }
  777. // flush the three tables' index/meta state to disk (after a run, and every few shows)
  778. static persistSync = () => {
  779. showsTable.persist()
  780. seasonsTable.persist()
  781. episodesTable.persist()
  782. return true
  783. }
  784. // totals of a run: the per-show counts added up
  785. static emptyTotals = () => { return { shows = 0 requests = 0 tvmazeRequests = 0 newSeasons = 0 newEpisodes = 0 updatedEpisodes = 0 updatedSeasons = 0 posters = 0 ids = 0 tvmazeEpisodes = 0 tvmazeSeasons = 0 tvmazeFilled = 0 tvmazeSkipped = 0 errors = 0 } }
  786. static addTotals = (t, r) => {
  787. let o = t
  788. o.shows = o.shows + 1
  789. o.requests = o.requests + r.requests
  790. o.tvmazeRequests = o.tvmazeRequests + r.tvmazeRequests
  791. o.ids = o.ids + r.ids
  792. o.newSeasons = o.newSeasons + r.newSeasons
  793. o.newEpisodes = o.newEpisodes + r.newEpisodes
  794. o.updatedEpisodes = o.updatedEpisodes + r.updatedEpisodes
  795. o.updatedSeasons = o.updatedSeasons + r.updatedSeasons
  796. o.posters = o.posters + r.posters
  797. o.tvmazeEpisodes = o.tvmazeEpisodes + r.tvmazeEpisodes
  798. o.tvmazeSeasons = o.tvmazeSeasons + r.tvmazeSeasons
  799. o.tvmazeFilled = o.tvmazeFilled + r.tvmazeFilled
  800. o.tvmazeSkipped = o.tvmazeSkipped + r.tvmazeSkipped
  801. o.errors = o.errors + r.errors
  802. return o
  803. }
  804. static totalsLine = (t) => {
  805. return 'shows=' + t.shows + ' newSeasons=' + t.newSeasons + ' newEpisodes=' + t.newEpisodes + ' updatedEpisodes=' + t.updatedEpisodes
  806. + ' updatedSeasons=' + t.updatedSeasons + ' posters=' + t.posters + ' ids=' + t.ids + ' requests=' + t.requests + ' tvmazeRequests=' + t.tvmazeRequests + ' errors=' + t.errors
  807. + ' tvmazeEpisodes=' + t.tvmazeEpisodes + ' tvmazeSeasons=' + t.tvmazeSeasons + ' tvmazeFilled=' + t.tvmazeFilled + ' tvmazeSkipped=' + t.tvmazeSkipped
  808. }

Branches

Latest commits

  • 46b21389tracker (mission 070, conductor): the sync never destroys migrated data — the one-time summary step MOVES a copied summary to migratedSummary (marker summaries-moved.txt) instead of clearing it; a migrated record's first title/genres/homepage/tagline TMDB replaces → migratedTitle/migratedGenres/migratedHomepage/migratedTagline, a migrated season's/episode's title/summary → migratedTitle/migratedSummary (set once); tests/peek-shows.hl prints them; gate 325/0mre
  • 749019b0Merge t19 (tracker#19 franchises + timelines) into main (mission 070): conflicts README/STATUS/show.hl/project.hl/browser.mjs/faketmdb.mjs, both sides kept; franchise/timeline pages get #20's typed heading (Franchise | …, Timeline | …), their title links via titlePath (/movies|/series|/shows); the collection seed waits for repair, kinds and credits too; franchises.mjs URLs + 2 new checks; gates browser 323/0, kinds 32/0, franchises 50/0mre
  • daf49feaMerge t20 (tracker#20 typed headings + #21 series/shows split) into main (mission 070): conflicts README/STATUS/show.hl/project.hl/search.hl/components/search.hl/browser.mjs/faketmdb.mjs, both sides kept; clock order backfill → repair → kinds → credits; withDetailsFields stores tmdbType + kind; gates: browser.mjs URLs → /series|/movies, typed h1 selectors; kinds.mjs repair/credits off, fixture name = seed name; browser 323/0, kinds 32/0mre
  • 20e09d89Merge t18 (tracker#18 delta sync) into main (mission 070): conflicts README/STATUS/show.hl/project.hl/browser.mjs, both sides kept; pageShowOf summary = summaryOfmre
  • f83571c3tracker#18 (mission 067): daily sync by change lists — TMDB /tv|movie/changes (since the stored day, paged) + TVmaze /updates/shows → only our changed titles (followed: full step, unfollowed: light step — changed seasons, no TVmaze), full walk on first run / gap > 14 days / failed list; show record refreshed (title, tmdbSummary, tagline, status, genres …; renamed titles re-indexed); summary = the creator's own text (page: summary > tmdbSummary > tvmazeSummary), one-time clear of copied summaries (9,647 on the live copy); gate 261, tests/realdata-018*.mjs, README + STATUSmre
  • 1ed5457etracker#20 + #21 (mission 068): typed headings "<Type> | <name>" in type colours; TV titles split into Series (/series) and Shows (/shows) by TMDB type + Reality/Talk/News genres — kind stored by sync/import/adult backfill + new kind backfill (resumes), /movies/<slug>, /shows/<slug> of a series/movie → 301, /my/series + /my/shows, home 5 tiles + 3 rows, search/filmography labels; gates kinds 32 + browser 266, tests/realdata-068.mjs, tools/count-kinds.hl, docs/kinds.md, README + STATUSmre
  • f2b00674Merge t26 (tracker#26 + #28) into main (mission 062): short ids for every new person (castPersonId, guest route), guest stars stored on the title and created as people only when opened (/person/tmdb/<id>?show=<id> → 302), lean watch/follow clicks (showRow a small object, cast/crew from the slug, watches cached per user, face rows only after a season toggle); gate 311, tests/realdata-062.mjs, README + STATUSmre
  • 0553b51ftracker#19 (mission 066): franchises and timelines — tables, /franchises, /franchises/<slug>, /timelines/<slug> (Timeline | Release sort, series by last episode), the Prequel | Timeline | Sequel widget with the franchise above, the creator's editor, TMDB collection seed in the app (resumes, paced); gate tests/franchises.mjs 48/0 + browser.mjs 266/0, tests/realdata-066.mjs, docs/franchises.md, README + STATUSmre
  • fa1f9dfatracker#29 (mission 063): unwatched check muted grey outline + check (accent only on hover), watched stays solid — no code regression, the accent outline read as ticked; gate checks real checks visibly (computed style + screenshot pixel) on /my/unwatched, show, movie, /my/movies; gate 266, tests/realdata-063.mjs, README + STATUSmre
  • 10bb3f93tracker#28 (mission 061): full cast (all seasons, main cast by episodes, guest stars) + crew (created by, directed by, written by, screenplay, story, music) — stored by the details completion, the daily sync, the search import (one details request) and a background credits job (resumes, RSS limit); show page collapsed after 20 with client-side Show all; showBySlug via a slug map; gate 249, tests/realdata-028.mjs, README + STATUSmre
  • d8b12d67tracker#27 (mission 060): short ids for movies, series and persons — old 702 kept (data/old-short-ids.json), new random [a-z0-9]{5} unique across both, claimed at creation, background backfill (resumes), shown under poster/photo, /<shortId> → 301; gate 259, tests/realdata-060*, README + STATUSmre
  • 25a50bc4tracker#26 (mission 059): titles from a filmography are completed — on open (skeleton, step-wise face showComplete, no reload) and by the in-app details repair (resumes, TMDB-paced, series in parts); cast from TMDB credits; gate 231, tests/realdata-026.mjs, README + STATUSmre
  • f14db671tracker#22-#25 (mission 058): episode air dates, season check = all episodes watched, movie watched check (+ /my/movies count), /genres/<genre> pages (movies + series, newest first, paginated); gate 238, tests/realdata-058.mjs, README + STATUSmre
  • 1704ec45tracker#17 (mission 057): season caret down/up, skeleton rows while a season loads, sessionless showSeasonEpisodes face (no page re-mount), client-only close; gate 214, tests/realdata-057.mjs, README + STATUSmre
  • f2fe3e36mission 056: README + STATUS (merge, fixes, Hybriel 8590df63, real-data check), tests/realdata-056.mjs, tools/check-public-slugs.hlmre
  • f40c250emission 056: re-vendor hybriel master 8590df63 (#121, #122); an adult title's page is Not found for non-followers; gate: leave the page before stopping the servermre
  • 2b7fdd6cmission 056: signed-out header one row on phones ("Log in", nowrap), backfill skips adult titles' posters, gate checksmre
  • 2c53d5efMerge branch 't16-person' (tracker#16 person pages) into main; filmography shows only public titles (054 adult flag), gate race fix (backfill start line)mre
  • c171227emission 054: hide adult/unknown titles from the public lists and the search; in-app adult-flag backfill (TMDB details + poster per title, resumes), gate + real-data proofmre
  • 139fafd8tracker#16: short bio (4 lines, click = all), real-data check script, README + STATUSmre