gitoriaLog in with ident

tracker

All repositories: gitoria

ReadmeCodePull requestsReleasesTicketsSettings
Commitf2b00674f2b00674Merge t26 (tracker#26 + #28) into main (mission 062): short ids for every new person (castPersonId, guest route), guest stars stored on the title and created as people only when opened (/person/tmdb/<id>?show=<id> → 302), lean watch/follow clicks (showRow a small object, cast/crew from the slug, watches cached per user, face rows only after a season toggle); gate 311, tests/realdata-062.mjs, README + STATUSmref2b00674/details.hl

23.2 KB

  1. // details.hl — A TITLE'S FULL DETAILS (tracker.worldapi.org#26). Creator: "when i go to an artist and it loads the bibliography
  2. // those loaded movies and series miss most information and it stays that way when i click on them".
  3. //
  4. // The person page's fill (people.hl, #16) adds every title we don't have as a MINIMAL record — title, year, overview, genres,
  5. // type, tmdbId, adult, this one person as cast; no poster, no seasons, no external ids, no homepage/tagline/status
  6. // (`minimal = true`). A search import (search.hl importTitle) is full except for the cast. Here a minimal title gets what a
  7. // full import has, plus its cast:
  8. //
  9. // GET <TMDB>/movie/<tmdbId>?append_to_response=external_ids,credits a movie
  10. // GET <TMDB>/tv/<tmdbId>?append_to_response=external_ids,aggregate_credits a series (cast of ALL seasons, roles merged)
  11. // then tmdbsync.hl `syncShowPart(id, <that answer>, …)` — the daily sync's work, without asking for the details again, IN
  12. // STEPS (see `jobs` below): a series' seasons + episodes (`season/1,…`), then the poster (w342), missing imdb/tvdb ids,
  13. // TVmaze lookup + its episode merge (#15); a movie its imdb id + poster
  14. //
  15. // Stored: overview (`summary` only when ours is empty; `tmdbSummary`), homepage, tagline, status, genres (by TMDB id, when
  16. // TMDB gives any), release/year/language when we have none, `adult`, the CAST (TMDB's first `castLimit`, in TMDB's order;
  17. // a person we have by TMDB id is linked, a new one is added — people.hl castPersonId; cast entries we had and TMDB does not
  18. // list are kept after them). Then `detailsAt` (ms) = COMPLETE, never asked again (the daily sync keeps a FOLLOWED title fresh).
  19. // TMDB 404 → `detailsCheck = 'TMDB 404'` (never asked again; the page shows what it has). TMDB says ADULT → the flag is
  20. // stored, `detailsCheck = 'adult'`, no poster/seasons (mission 056: an adult title is never public, its poster not kept).
  21. //
  22. // WHO CALLS IT (`completeStep`, again and again until `done`): the show page when an incomplete title is opened
  23. // (components/show.hl face `showComplete`, skeleton meanwhile) and the repair job in project.hl (`repairTick`, one step per tick
  24. // on the app's clock, like the adult backfill) for every incomplete title that exists when the app starts. Adult titles are
  25. // left alone by both.
  26. //
  27. // CAST AND CREW (tracker.worldapi.org#28; creator: "you show quite few people, usually there are more especially at picard, and
  28. // directors and so"). From the SAME details answer (`credits` of a movie, `aggregate_credits` of a series = every season):
  29. // cast [{ person, character, actor, episodes, guest }] — a movie: TMDB's order (billing); a series: TMDB's aggregate order
  30. // (by episode count, most first; `character` = all roles " / "), `episodes` = the person's episodes, `guest` = fewer than
  31. // `mainMinOf` (half the series' episodes, at most 10, at least 1) → the page puts them after "Guest stars". At most
  32. // `castMax` (300: a talk show/soap lists 3,000+ one-episode guests — Grey's Anatomy 3,243, SNL 2,771 — each would be a
  33. // new person); adult people left out; entries we had and TMDB does not list are kept after them.
  34. // GUESTS ARE NOT PERSON RECORDS (mission 062, memory: 160k people = +3.3 GB RSS at boot): a guest entry is
  35. // { person (only when we have them already, else null), tmdbId, actor, character, profile (TMDB's profile_path), episodes,
  36. // guest = true } — the page links it to `/person/tmdb/<tmdbId>?show=<show id>`, which creates the person on the first open
  37. // (people.hl guestPersonId). The main cast and the crew are linked/created at once as before.
  38. // crew [{ person, name, job, episodes }] job = creator (a series' `created_by`) | director (Director) | writer (Writer) |
  39. // screenplay (Screenplay) | story (Story) | composer (Original Music Composer, Composer, Music) — one entry per person and
  40. // job, a series' by that job's episode count (most first), a movie's in TMDB's order; at most `crewMax` per job.
  41. // `creditsAt` (ms) = the credits are stored (by any path); `creditsCheck` = why the credits job gave up (TMDB 404, adult).
  42. // Every person is linked (people.hl castPersonId: the one we have by TMDB id, else a NEW minimal person — slug, search entry,
  43. // this one credit; their page fills on the first visit). WHO STORES THEM: the details step below (incomplete titles), the daily
  44. // sync and the search import (tmdbsync.hl asks for the credits in the details request it makes anyway → `applyCredits`), and
  45. // the CREDITS JOB (project.hl `creditsTick`, `creditsStep` here) for every other public title once — one request per title.
  46. import { now } from 'hl:time'
  47. import { env } from 'hl:proc'
  48. import { showsTable, personsTable } from './shows.hl'
  49. import { tmdbGet, textOr, adultOf, syncShowWith, syncShowPart, persistSync } from './tmdbsync.hl'
  50. import { searchTitleChanged } from './search.hl'
  51. import { refreshCatalogShow } from './catalog.hl'
  52. import { castPersonId, castPeopleCount, genreIdsOfTmdb, personIdByTmdb } from './people.hl'
  53. // how many of TMDB's cast a title keeps (see the top; TRACKER_CAST_MAX) and how many people per crew job
  54. static castMax = env('TRACKER_CAST_MAX') != null ? toNumber(env('TRACKER_CAST_MAX')) : 300
  55. static crewMax = 30
  56. static listOr = (v) => { return v == null ? [] : v }
  57. // INCOMPLETE = a filmography title (`minimal`) or an import whose sync never finished (`imported`, no `tmdbSync`), not
  58. // completed (`detailsAt`) and not given up on (`detailsCheck`)
  59. static isIncomplete = (s) => {
  60. return s != null && s.tmdbId != null && s.detailsAt == null && s.detailsCheck == null && (s.minimal == true || (s.imported != null && s.tmdbSync == null))
  61. }
  62. // what the show page and the repair complete: an incomplete title that is not adult (mission 054/056: never public)
  63. static needsDetails = (s) => { return isIncomplete(s) && s.adult != true }
  64. // every title the repair job has to do, in table order (read once when it starts — so a restart goes on with what is left)
  65. static detailsRepairQueue = () => {
  66. let out = []
  67. for (s of showsTable.find(null, null)) { if (needsDetails(s)) { out.push(s.id) } }
  68. return out
  69. }
  70. // a series' roles ([{ character, episode_count }]) as one text: "Mother / Lamia". tracker#28 (real data): TMDB's characters are
  71. // themselves " / "-joined and repeat across seasons ("Data / Lore", "Data", "Yvette Picard (uncredited)") — split, " (uncredited)"
  72. // dropped, each name once, at most `rolesMax` (a voice actor of The Simpsons has 30+), then " / …"
  73. static rolesMax = 4
  74. static rolesText = (roles) => {
  75. let names = []
  76. let more = false
  77. for (r of listOr(roles)) {
  78. let c = r != null ? textOr(r.character) : null
  79. if (c != null) {
  80. for (part of c.split(' / ')) {
  81. let n = part.replaceAll(' (uncredited)', '').trim()
  82. if (n != '' && !names.includes(n)) { if (names.length < rolesMax) { names.push(n) } else { more = true } }
  83. }
  84. }
  85. }
  86. return names.join(' / ') + (more ? ' / …' : '')
  87. }
  88. // a series' episodes of one person: TMDB's total_episode_count, else the sum of the roles' / jobs' counts
  89. static episodesOf = (total, parts) => {
  90. if (total != null && hlTypeName(total) == 'Number') { return total }
  91. let n = 0
  92. for (r of listOr(parts)) { if (r != null && r.episode_count != null && hlTypeName(r.episode_count) == 'Number') { n = n + r.episode_count } }
  93. return n
  94. }
  95. static personOk = (c) => { return c != null && c.id != null && hlTypeName(c.id) == 'Number' && textOr(c.name) != null && c.adult != true }
  96. // TMDB's cast → [{ tmdbId, name, character, gender, episodes }], TMDB's order, at most castMax, adult people left out
  97. static castOf = (d, movie) => {
  98. let src = movie ? (d.credits != null ? d.credits.cast : null) : (d.aggregate_credits != null ? d.aggregate_credits.cast : null)
  99. let out = []
  100. for (c of listOr(src)) {
  101. if (out.length < castMax && personOk(c)) {
  102. let role = movie ? textOr(c.character) : rolesText(c.roles)
  103. out.push({ tmdbId = c.id name = c.name character = role != null ? role : '' gender = c.gender profile = textOr(c.profile_path) episodes = movie ? null : episodesOf(c.total_episode_count, c.roles) })
  104. }
  105. }
  106. return out
  107. }
  108. // a series' main cast is in at least this many episodes: half of the series' (rounded up), at most 10 (a season), at least 1
  109. static mainMinOf = (d) => {
  110. let n = d.number_of_episodes != null && hlTypeName(d.number_of_episodes) == 'Number' ? d.number_of_episodes : 0
  111. let half = (n - n % 2) / 2 + n % 2
  112. if (half > 10) { return 10 }
  113. return half < 1 ? 1 : half
  114. }
  115. // the crew jobs we show: TMDB job → ours (the page's labels: components/show.hl crewLabels)
  116. static crewJobOf = (job) => {
  117. if (job == 'Director') { return 'director' }
  118. if (job == 'Writer') { return 'writer' }
  119. if (job == 'Screenplay') { return 'screenplay' }
  120. if (job == 'Story') { return 'story' }
  121. if (job == 'Original Music Composer' || job == 'Composer' || job == 'Music') { return 'composer' }
  122. return null
  123. }
  124. static crewJobs = ['creator', 'director', 'writer', 'screenplay', 'story', 'composer']
  125. // keeps a job's list ordered by episodes (most first, equal ones in TMDB's order), at most crewMax (hand-written: no sort())
  126. static insertByEpisodes = (list, item) => {
  127. let out = []
  128. let inserted = false
  129. for (x of list) {
  130. if (!inserted && item.episodes > x.episodes) { out.push(item) inserted = true }
  131. if (out.length < crewMax) { out.push(x) }
  132. }
  133. if (!inserted && out.length < crewMax) { out.push(item) }
  134. return out
  135. }
  136. // TMDB's crew (+ a series' created_by) → [{ tmdbId, name, job, gender, episodes }], grouped by job in `crewJobs` order, one
  137. // entry per person and job
  138. static crewOf = (d, movie) => {
  139. let byJob = {}
  140. let seen = {}
  141. for (j of crewJobs) { byJob[j] = [] }
  142. if (!movie) {
  143. for (c of listOr(d.created_by)) {
  144. if (personOk(c) && seen['creator:' + c.id] != true && byJob['creator'].length < crewMax) {
  145. seen['creator:' + c.id] = true
  146. let l = byJob['creator']
  147. l.push({ tmdbId = c.id name = c.name job = 'creator' gender = c.gender episodes = 0 })
  148. byJob['creator'] = l
  149. }
  150. }
  151. }
  152. let src = movie ? (d.credits != null ? d.credits.crew : null) : (d.aggregate_credits != null ? d.aggregate_credits.crew : null)
  153. for (c of listOr(src)) {
  154. if (personOk(c)) {
  155. // a movie's crew entry has one `job`; a series' has `jobs` [{ job, episode_count }]
  156. let jobs = movie ? [{ job = c.job episode_count = 0 }] : listOr(c.jobs)
  157. for (jb of jobs) {
  158. let ours = jb != null ? crewJobOf(jb.job) : null
  159. if (ours != null && seen[ours + ':' + c.id] != true) {
  160. seen[ours + ':' + c.id] = true
  161. let n = movie ? 0 : episodesOf(jb.episode_count, null)
  162. byJob[ours] = insertByEpisodes(byJob[ours], { tmdbId = c.id name = c.name job = ours gender = c.gender episodes = n })
  163. }
  164. }
  165. }
  166. }
  167. let out = []
  168. for (j of crewJobs) { for (c of byJob[j]) { out.push(c) } }
  169. return out
  170. }
  171. // the title's cast: TMDB's (linked to our people, new ones added) first, then the entries we had that TMDB does not list.
  172. // `mainMin` = a series' main-cast episode count (mainMinOf), 0 for a movie (no guests)
  173. static mergedCast = (show, tmdbCast, mainMin) => {
  174. let out = []
  175. let have = {}
  176. let showKey = show.id
  177. let seenTmdb = {}
  178. for (c of tmdbCast) {
  179. let guest = c.episodes != null && c.episodes < mainMin
  180. // a guest: linked only when we have them, never created here (mission 062)
  181. let pid = guest ? personIdByTmdb(c.tmdbId) : castPersonId(c, showKey)
  182. if (pid != null && have[pid] != true) {
  183. have[pid] = true
  184. seenTmdb['' + c.tmdbId] = true
  185. out.push({ person = pid tmdbId = c.tmdbId character = c.character actor = c.name profile = c.profile episodes = c.episodes guest = guest })
  186. } else if (pid == null && guest && seenTmdb['' + c.tmdbId] != true) {
  187. seenTmdb['' + c.tmdbId] = true
  188. out.push({ person = null tmdbId = c.tmdbId character = c.character actor = c.name profile = c.profile episodes = c.episodes guest = true })
  189. }
  190. }
  191. for (c of listOr(show.cast)) {
  192. if (c != null && c.person != null && have[c.person] != true && (c.tmdbId == null || seenTmdb['' + c.tmdbId] != true)) {
  193. have[c.person] = true
  194. out.push(c)
  195. }
  196. }
  197. return out
  198. }
  199. static crewRoleOf = (job) => {
  200. if (job == 'creator') { return 'Creator' }
  201. if (job == 'director') { return 'Director' }
  202. if (job == 'writer') { return 'Writer' }
  203. if (job == 'screenplay') { return 'Screenplay' }
  204. if (job == 'story') { return 'Story' }
  205. return 'Music'
  206. }
  207. // the title's crew: TMDB's, linked to our people (a crew person new to us gets the job as their role) — replaces what we had
  208. static linkedCrew = (show, tmdbCrew) => {
  209. let out = []
  210. let showKey = show.id
  211. for (c of tmdbCrew) {
  212. let pid = castPersonId({ tmdbId = c.tmdbId name = c.name character = crewRoleOf(c.job) gender = c.gender }, showKey)
  213. if (pid != null) { out.push({ person = pid name = c.name job = c.job episodes = c.episodes }) }
  214. }
  215. return out
  216. }
  217. // the answer carries credits (a details request with `credits` / `aggregate_credits` appended)
  218. static hasCredits = (d, movie) => { return d != null && (movie ? d.credits != null : d.aggregate_credits != null) }
  219. // cast + crew of this record from a details answer → { show (the record with cast, crew, creditsAt), cast, crew, people (new) }
  220. static withCredits = (show, d, movie) => {
  221. let before = castPeopleCount()
  222. let tmdbCast = castOf(d, movie)
  223. let x = show
  224. x.cast = mergedCast(show, tmdbCast, movie ? 0 : mainMinOf(d))
  225. x.crew = linkedCrew(show, crewOf(d, movie))
  226. x.creditsAt = now()
  227. return { show = x cast = tmdbCast.length crew = x.crew.length people = castPeopleCount() - before }
  228. }
  229. // the daily sync and the search import: their details answer (tmdbsync.hl `res.details`) → the title's cast + crew. Nothing
  230. // for an adult title or an answer without credits. Answers { cast, crew, people, ms, error }
  231. static applyCredits = (showId, d) => {
  232. let out = { cast = 0 crew = 0 people = 0 ms = 0 error = '' }
  233. let t0 = now()
  234. let show = showsTable.fetch(showId)
  235. if (show == null || show.adult == true || adultOf(d) == true || !hasCredits(d, show.type == 'movie')) { return out }
  236. let w = withCredits(show, d, show.type == 'movie')
  237. if (showsTable.update(show.id, w.show) == null) { out.error = show.title + ': credits not stored: ' + showsTable.lastError() return out }
  238. out.cast = w.cast
  239. out.crew = w.crew
  240. out.people = w.people
  241. out.ms = now() - t0
  242. return out
  243. }
  244. // A SERIES IS DONE IN STEPS (real data: The Late Show, 22 seasons / 4,252 episodes, took 10.7 s as one step — hl:web serves one
  245. // request at a time, every page waited): step 1 = the details request (fields + cast); then one step per PART of its seasons
  246. // (≤ `partEpisodes` episodes, ≤ 20 seasons = one TMDB request — tmdbsync.hl syncShowPart); the last step = ids, TVmaze, poster,
  247. // `detailsAt`. A movie is one step. Between two steps every other request is served.
  248. // show id → { d (TMDB's answer of step 1), parts ([[season numbers]]), pos, sum } while a series is under way — in memory: a
  249. // restart begins again with the details request; what is stored by then is matched by tmdbId, nothing is duplicated
  250. static jobs = {}
  251. // (TRACKER_DETAILS_PART_EPISODES: the gate sets 1 → one season per step)
  252. static partEpisodes = env('TRACKER_DETAILS_PART_EPISODES') != null ? toNumber(env('TRACKER_DETAILS_PART_EPISODES')) : 300
  253. static emptySum = () => { return { steps = 0 requests = 0 tvmazeRequests = 0 cast = 0 crew = 0 people = 0 newSeasons = 0 newEpisodes = 0 posters = 0 ids = 0 ms = 0 error = '' } }
  254. static addSum = (sum, r, ms) => {
  255. let o = sum
  256. o.steps = o.steps + 1
  257. o.requests = o.requests + r.requests
  258. o.tvmazeRequests = o.tvmazeRequests + r.tvmazeRequests
  259. o.newSeasons = o.newSeasons + r.newSeasons + r.tvmazeSeasons
  260. o.newEpisodes = o.newEpisodes + r.newEpisodes + r.tvmazeEpisodes
  261. o.posters = o.posters + r.posters
  262. o.ids = o.ids + r.ids
  263. o.ms = o.ms + ms
  264. if (r.error != '') { o.error = r.error }
  265. return o
  266. }
  267. // TMDB's season list → parts of ≤ partEpisodes episodes (a bigger season is a part of its own), ≤ 20 seasons each; which of
  268. // them are really synced (season 0, empty ones) syncShowPart decides
  269. static seasonParts = (d) => {
  270. let parts = []
  271. let cur = []
  272. let n = 0
  273. for (x of listOr(d.seasons)) {
  274. if (x != null && x.season_number != null) {
  275. let c = x.episode_count != null ? x.episode_count : 0
  276. if (cur.length > 0 && (n + c > partEpisodes || cur.length == 20)) {
  277. parts.push(cur)
  278. cur = []
  279. n = 0
  280. }
  281. cur.push(x.season_number)
  282. n = n + c
  283. }
  284. }
  285. if (cur.length > 0) { parts.push(cur) }
  286. return parts
  287. }
  288. // a step's answer: { skipped (nothing to do), done, requests / tvmazeRequests (THIS step's — what the pause counts), adult,
  289. // error ('' = none), retry (a passing failure: no answer, 429, 5xx), sum (the whole title's counts so far: steps, requests,
  290. // cast, newSeasons, newEpisodes, posters, ids, ms) }
  291. static stepOut = () => { return { skipped = false done = false requests = 0 tvmazeRequests = 0 adult = null error = '' retry = false sum = emptySum() } }
  292. // the last step: the title is complete
  293. static markComplete = (showId, sum) => {
  294. let done = showsTable.fetch(showId)
  295. done.detailsAt = now()
  296. if (sum.error != '') { done.detailsError = sum.error }
  297. let ok = showsTable.update(done.id, done) != null
  298. refreshCatalogShow(done.id)
  299. searchTitleChanged(done.id)
  300. return ok
  301. }
  302. // step 1: the details request — the record's fields, the cast; a movie also its ids + poster (done); a series' parts are planned
  303. static firstStep = (show) => {
  304. let out = stepOut()
  305. let t0 = now()
  306. let movie = show.type == 'movie'
  307. out.requests = 1
  308. let d = tmdbGet((movie ? '/movie/' : '/tv/') + show.tmdbId + '?append_to_response=external_ids,' + (movie ? 'credits' : 'aggregate_credits'))
  309. if (d.failed != null) {
  310. out.error = show.title + ': ' + d.failed
  311. if (d.status == 404) {
  312. show.detailsCheck = 'TMDB 404'
  313. showsTable.update(show.id, show)
  314. out.done = true
  315. } else { out.retry = true }
  316. return out
  317. }
  318. let x = show
  319. out.adult = adultOf(d)
  320. if (out.adult != null) { x.adult = out.adult }
  321. let overview = textOr(d.overview)
  322. if (overview != null) {
  323. x.tmdbSummary = overview
  324. if (textOr(x.summary) == null) { x.summary = overview }
  325. }
  326. if (textOr(d.homepage) != null) { x.homepage = d.homepage }
  327. if (textOr(d.tagline) != null) { x.tagline = d.tagline }
  328. if (textOr(d.status) != null) { x.status = d.status }
  329. let tmdbGenreIds = []
  330. for (g of listOr(d.genres)) { if (g != null && g.id != null) { tmdbGenreIds.push(g.id) } }
  331. let genres = genreIdsOfTmdb(tmdbGenreIds)
  332. if (genres.length > 0) { x.genres = genres }
  333. let release = textOr(movie ? d.release_date : d.first_air_date)
  334. if (textOr(x.release) == null && release != null) {
  335. x.release = release
  336. if (release.length >= 4) { x.year = toNumber(release.slice(0, 4)) }
  337. }
  338. if (textOr(x.language) == null && textOr(d.original_language) != null) { x.language = d.original_language }
  339. if (out.adult == true) {
  340. // never public: the flag, no poster, no seasons, no cast (mission 056)
  341. x.detailsCheck = 'adult'
  342. showsTable.update(x.id, x)
  343. refreshCatalogShow(x.id)
  344. searchTitleChanged(x.id)
  345. out.done = true
  346. return out
  347. }
  348. // tracker#28: the whole cast + the crew, from this same answer
  349. let w = withCredits(x, d, movie)
  350. x = w.show
  351. if (showsTable.update(x.id, x) == null) {
  352. out.error = show.title + ': not updated: ' + showsTable.lastError()
  353. out.retry = true
  354. return out
  355. }
  356. let sum = emptySum()
  357. sum.cast = w.cast
  358. sum.crew = w.crew
  359. sum.people = w.people
  360. if (movie) {
  361. // one request was enough: ids + poster now (syncMovie with this answer)
  362. let r = syncShowWith(x.id, d)
  363. out.requests = r.requests
  364. out.sum = addSum(sum, r, now() - t0)
  365. if (r.error != '') { out.error = r.error }
  366. markComplete(x.id, out.sum)
  367. out.done = true
  368. return out
  369. }
  370. sum.steps = 1
  371. sum.requests = 1
  372. sum.ms = now() - t0
  373. // the next steps need the details, not the credits (a long series' aggregate_credits are ~1 MB, copied on every step)
  374. let dd = d
  375. dd.aggregate_credits = null
  376. jobs[x.id] = { d = dd parts = seasonParts(d) pos = 0 sum = sum }
  377. out.sum = sum
  378. return out
  379. }
  380. // ONE step of completing this title (by our id) — called until `done` (the page's face, the repair job)
  381. static completeStep = (showId) => {
  382. let show = showsTable.fetch(showId)
  383. if (!needsDetails(show)) {
  384. jobs[showId] = null
  385. let out = stepOut()
  386. out.skipped = true
  387. out.done = true
  388. return out
  389. }
  390. let job = jobs[showId]
  391. if (job == null) { return firstStep(show) }
  392. let t0 = now()
  393. let out = stepOut()
  394. let last = job.pos >= job.parts.length
  395. let r = syncShowPart(showId, job.d, { seasons = last ? [] : job.parts[job.pos] finish = last })
  396. out.requests = r.requests
  397. out.tvmazeRequests = r.tvmazeRequests
  398. if (r.error != '') { out.error = r.error }
  399. let j = job
  400. j.sum = addSum(j.sum, r, now() - t0)
  401. j.pos = j.pos + 1
  402. out.sum = j.sum
  403. if (!last) {
  404. jobs[showId] = j
  405. return out
  406. }
  407. jobs[showId] = null
  408. markComplete(showId, j.sum)
  409. out.done = true
  410. return out
  411. }
  412. // the line every completed title logs
  413. static completeLine = (title, sum) => {
  414. return '"' + title + '" steps=' + sum.steps + ' requests=' + sum.requests + ' tvmazeRequests=' + sum.tvmazeRequests + ' cast=' + sum.cast + ' newSeasons=' + sum.newSeasons
  415. + ' newEpisodes=' + sum.newEpisodes + ' posters=' + sum.posters + ' ids=' + sum.ids + ' ms=' + sum.ms + ' crew=' + sum.crew + ' people=' + sum.people + (sum.error != '' ? ' error: ' + sum.error : '')
  416. }
  417. // flush shows/seasons/episodes + people (new cast members)
  418. static persistDetails = () => {
  419. persistSync()
  420. personsTable.persist()
  421. return true
  422. }
  423. // the credits job touches only shows + people: the episodes' index files (~16 MB on the live data) are not rewritten
  424. static persistCredits = () => {
  425. showsTable.persist()
  426. personsTable.persist()
  427. return true
  428. }
  429. // ---- THE CREDITS JOB (tracker#28, project.hl `creditsTick`): every public title that has no credits yet, once --------------
  430. // (the incomplete ones get theirs from completeStep; adult and unknown-flag titles are never asked — not public)
  431. static needsCredits = (s) => { return s != null && s.tmdbId != null && s.adult == false && s.creditsAt == null && s.creditsCheck == null && !isIncomplete(s) }
  432. // in table order, read once when the job starts (a restart goes on with what is left)
  433. static creditsQueue = () => {
  434. let out = []
  435. for (s of showsTable.find(null, null)) { if (needsCredits(s)) { out.push(s.id) } }
  436. return out
  437. }
  438. // ONE title: one request (details + credits / aggregate_credits) → its cast + crew. Answers { skipped, requests, cast, crew,
  439. // people, ms, error, retry (no answer, 429, 5xx: ask again later) }. TMDB 404 / adult → `creditsCheck`, never asked again.
  440. static creditsStep = (showId) => {
  441. let out = { skipped = false requests = 0 cast = 0 crew = 0 people = 0 ms = 0 error = '' retry = false }
  442. let t0 = now()
  443. let show = showsTable.fetch(showId)
  444. if (!needsCredits(show)) { out.skipped = true return out }
  445. let movie = show.type == 'movie'
  446. out.requests = 1
  447. let d = tmdbGet((movie ? '/movie/' : '/tv/') + show.tmdbId + '?append_to_response=' + (movie ? 'credits' : 'aggregate_credits'))
  448. if (d.failed != null) {
  449. out.error = show.title + ': ' + d.failed
  450. if (d.status == 404) {
  451. show.creditsCheck = 'TMDB 404'
  452. showsTable.update(show.id, show)
  453. } else { out.retry = true }
  454. return out
  455. }
  456. if (adultOf(d) == true) {
  457. show.creditsCheck = 'adult'
  458. showsTable.update(show.id, show)
  459. return out
  460. }
  461. let w = withCredits(show, d, movie)
  462. if (showsTable.update(show.id, w.show) == null) {
  463. out.error = show.title + ': credits not stored: ' + showsTable.lastError()
  464. out.retry = true
  465. return out
  466. }
  467. out.cast = w.cast
  468. out.crew = w.crew
  469. out.people = w.people
  470. out.ms = now() - t0
  471. return out
  472. }

Branches

Latest commits

  • f2b00674Merge t26 (tracker#26 + #28) into main (mission 062): short ids for every new person (castPersonId, guest route), guest stars stored on the title and created as people only when opened (/person/tmdb/<id>?show=<id> → 302), lean watch/follow clicks (showRow a small object, cast/crew from the slug, watches cached per user, face rows only after a season toggle); gate 311, tests/realdata-062.mjs, README + STATUSmre
  • fa1f9dfatracker#29 (mission 063): unwatched check muted grey outline + check (accent only on hover), watched stays solid — no code regression, the accent outline read as ticked; gate checks real checks visibly (computed style + screenshot pixel) on /my/unwatched, show, movie, /my/movies; gate 266, tests/realdata-063.mjs, README + STATUSmre
  • 10bb3f93tracker#28 (mission 061): full cast (all seasons, main cast by episodes, guest stars) + crew (created by, directed by, written by, screenplay, story, music) — stored by the details completion, the daily sync, the search import (one details request) and a background credits job (resumes, RSS limit); show page collapsed after 20 with client-side Show all; showBySlug via a slug map; gate 249, tests/realdata-028.mjs, README + STATUSmre
  • d8b12d67tracker#27 (mission 060): short ids for movies, series and persons — old 702 kept (data/old-short-ids.json), new random [a-z0-9]{5} unique across both, claimed at creation, background backfill (resumes), shown under poster/photo, /<shortId> → 301; gate 259, tests/realdata-060*, README + STATUSmre
  • 25a50bc4tracker#26 (mission 059): titles from a filmography are completed — on open (skeleton, step-wise face showComplete, no reload) and by the in-app details repair (resumes, TMDB-paced, series in parts); cast from TMDB credits; gate 231, tests/realdata-026.mjs, README + STATUSmre
  • f14db671tracker#22-#25 (mission 058): episode air dates, season check = all episodes watched, movie watched check (+ /my/movies count), /genres/<genre> pages (movies + series, newest first, paginated); gate 238, tests/realdata-058.mjs, README + STATUSmre
  • 1704ec45tracker#17 (mission 057): season caret down/up, skeleton rows while a season loads, sessionless showSeasonEpisodes face (no page re-mount), client-only close; gate 214, tests/realdata-057.mjs, README + STATUSmre
  • f2fe3e36mission 056: README + STATUS (merge, fixes, Hybriel 8590df63, real-data check), tests/realdata-056.mjs, tools/check-public-slugs.hlmre
  • f40c250emission 056: re-vendor hybriel master 8590df63 (#121, #122); an adult title's page is Not found for non-followers; gate: leave the page before stopping the servermre
  • 2b7fdd6cmission 056: signed-out header one row on phones ("Log in", nowrap), backfill skips adult titles' posters, gate checksmre
  • 2c53d5efMerge branch 't16-person' (tracker#16 person pages) into main; filmography shows only public titles (054 adult flag), gate race fix (backfill start line)mre
  • c171227emission 054: hide adult/unknown titles from the public lists and the search; in-app adult-flag backfill (TMDB details + poster per title, resumes), gate + real-data proofmre
  • 139fafd8tracker#16: short bio (4 lines, click = all), real-data check script, README + STATUSmre
  • 93be9476tracker#16: person pages /person/<slug> with the filmography fetched from TMDB on the first visit (step by step), gatemre
  • 47a3cae6STATUS: mission 053 merge commit idsmre
  • dcc5eecaMerge branch 't14-search'mre
  • 03edc783Merge branch 't15-tvmaze'mre
  • 71b46345tracker#15: numbering check by date or title, placeholder titles in other languages, docs + real-data proofmre
  • 6bb2daf1tracker#13: homepage (tiles, intro, latest movies/shows), /shows, /movies/page/N, /my/movies; lists cached in memorymre
  • b8bd1157tracker#14: README + STATUS (search, real-data numbers, gate, merge notes)mre