tracker

28.0 KB

1. // details.hl — A TITLE'S FULL DETAILS (tracker.worldapi.org#26). Creator: "when i go to an artist and it loads the bibliography
2. // those loaded movies and series miss most information and it stays that way when i click on them".
3. //
4. // The person page's fill (people.hl, #16) adds every title we don't have as a MINIMAL record — title, year, overview, genres,
5. // type, tmdbId, adult, this one person as cast; no poster, no seasons, no external ids, no homepage/tagline/status
6. // (`minimal = true`). A search import (search.hl importTitle) is full except for the cast. Here a minimal title gets what a
7. // full import has, plus its cast:
8. //
9. // GET <TMDB>/movie/<tmdbId>?append_to_response=external_ids,credits a movie
10. // GET <TMDB>/tv/<tmdbId>?append_to_response=external_ids,aggregate_credits a series (cast of ALL seasons, roles merged)
11. // then tmdbsync.hl `syncShowPart(id, <that answer>, …)` — the daily sync's work, without asking for the details again, IN
12. // STEPS (see `jobs` below): a series' seasons + episodes (`season/1,…`), then the poster (w342), missing imdb/tvdb ids,
13. // TVmaze lookup + its episode merge (#15); a movie its imdb id + poster
14. //
15. // Stored (`withDetailsFields`, shared with the daily sync since tracker#18): title, overview → `tmdbSummary` (`summary` is the
16. // creator's own text, never written), homepage, tagline, status, genres (by TMDB id, when
17. // TMDB gives any), release/year/language when we have none, `adult`, the CAST (TMDB's first `castLimit`, in TMDB's order;
18. // a person we have by TMDB id is linked, a new one is added — people.hl castPersonId; cast entries we had and TMDB does not
19. // list are kept after them). Then `detailsAt` (ms) = COMPLETE, never asked again (the daily sync keeps a FOLLOWED title fresh).
20. // TMDB 404 → `detailsCheck = 'TMDB 404'` (never asked again; the page shows what it has). TMDB says ADULT → the flag is
21. // stored, `detailsCheck = 'adult'`, no poster/seasons (mission 056: an adult title is never public, its poster not kept).
22. //
23. // WHO CALLS IT (`completeStep`, again and again until `done`): the show page when an incomplete title is opened
24. // (components/show.hl face `showComplete`, skeleton meanwhile) and the repair job in project.hl (`repairTick`, one step per tick
25. // on the app's clock, like the adult backfill) for every incomplete title that exists when the app starts. Adult titles are
26. // left alone by both.
27. //
28. // CAST AND CREW (tracker.worldapi.org#28; creator: "you show quite few people, usually there are more especially at picard, and
29. // directors and so"). From the SAME details answer (`credits` of a movie, `aggregate_credits` of a series = every season):
30. // cast [{ person, character, actor, episodes, guest }] — a movie: TMDB's order (billing); a series: TMDB's aggregate order
31. // (by episode count, most first; `character` = all roles " / "), `episodes` = the person's episodes, `guest` = fewer than
32. // `mainMinOf` (half the series' episodes, at most 10, at least 1) → the page puts them after "Guest stars". At most
33. // `castMax` (300: a talk show/soap lists 3,000+ one-episode guests — Grey's Anatomy 3,243, SNL 2,771 — each would be a
34. // new person); adult people left out; entries we had and TMDB does not list are kept after them.
35. // GUESTS ARE NOT PERSON RECORDS (mission 062, memory: 160k people = +3.3 GB RSS at boot): a guest entry is
36. // { person (only when we have them already, else null), tmdbId, actor, character, profile (TMDB's profile_path), episodes,
37. // guest = true } — the page links it to `/person/tmdb/<tmdbId>?show=<show id>`, which creates the person on the first open
38. // (people.hl guestPersonId). The main cast and the crew are linked/created at once as before.
39. // crew [{ person, name, job, episodes }] job = creator (a series' `created_by`) | director (Director) | writer (Writer) |
40. // screenplay (Screenplay) | story (Story) | composer (Original Music Composer, Composer, Music) — one entry per person and
41. // job, a series' by that job's episode count (most first), a movie's in TMDB's order; at most `crewMax` per job.
42. // `creditsAt` (ms) = the credits are stored (by any path); `creditsCheck` = why the credits job gave up (TMDB 404, adult).
43. // Every person is linked (people.hl castPersonId: the one we have by TMDB id, else a NEW minimal person — slug, search entry,
44. // this one credit; their page fills on the first visit). WHO STORES THEM: the details step below (incomplete titles), the daily
45. // sync and the search import (tmdbsync.hl asks for the credits in the details request it makes anyway → `applyCredits`), and
46. // the CREDITS JOB (project.hl `creditsTick`, `creditsStep` here) for every other public title once — one request per title.
47. import { now } from 'hl:time'
48. import { env } from 'hl:proc'
49. import { showsTable, personsTable, kindOfTitle } from './shows.hl'
50. import { tmdbGet, textOr, adultOf, syncShowWith, syncShowPart, persistSync } from './tmdbsync.hl'
51. import { searchTitleChanged } from './search.hl'
52. import { refreshCatalogShow } from './catalog.hl'
53. import { castPersonId, castPeopleCount, genreIdsOfTmdb, personIdByTmdb } from './people.hl'
55. // how many of TMDB's cast a title keeps (see the top; TRACKER_CAST_MAX) and how many people per crew job
56. static castMax = env('TRACKER_CAST_MAX') != null ? toNumber(env('TRACKER_CAST_MAX')) : 300
57. static crewMax = 30
59. static listOr = (&v) => { return v == null ? [] : v }
61. // INCOMPLETE = a filmography title (`minimal`) or an import whose sync never finished (`imported`, no `tmdbSync`), not
62. // completed (`detailsAt`) and not given up on (`detailsCheck`) — or a movie with a TV answer's leftovers (below)
63. static isIncomplete = (&s) => {
64. return s != null && s.tmdbId != null && s.detailsCheck == null
65. && ((s.detailsAt == null && (s.minimal == true || (s.imported != null && s.tmdbSync == null))) || hasTvLeftovers(s))
66. }
67. // mission 025 (report 021's "wrong TMDB id"): the #9 sync once asked TMDB for every followed title as `/tv/<tmdbId>` — a movie's
68. // id is ANOTHER title there (fixed in mission 006: movies go to /movie/) and the TV title's overview, status and season/episode
69. // counts were stored on the movie. On the real data only Star Trek: First Contact: its tmdbId 199 IS the movie (/movie/199),
70. // tmdbSummary was TV 199's ("Kabuhayang Swak na Swak …"), status 'Ended', seasonsCount/episodesCount 0. The movie path never
71. // writes the counts, so a movie that has them is such a record: the repair asks /movie/<id> again (withDetailsFields: overview →
72. // tmdbSummary, status, …) and firstStep drops the counts — repaired once, then it no longer matches.
73. static hasTvLeftovers = (&s) => { return s.type == 'movie' && (s.seasonsCount != null || s.episodesCount != null) }
74. // what the show page and the repair complete: an incomplete title that is not adult (mission 054/056: never public)
75. static needsDetails = (&s) => { return isIncomplete(s) && s.adult != true && s.mergedInto == null }
77. // every title the repair job has to do, in table order (read once when it starts — so a restart goes on with what is left)
78. static detailsRepairQueue = () => {
79. let out = []
80. for (s of showsTable.find(null, null)) { if (needsDetails(s)) { out.push(s.id) } }
81. return out
82. }
84. // a series' roles ([{ character, episode_count }]) as one text: "Mother / Lamia". tracker#28 (real data): TMDB's characters are
85. // themselves " / "-joined and repeat across seasons ("Data / Lore", "Data", "Yvette Picard (uncredited)") — split, " (uncredited)"
86. // dropped, each name once, at most `rolesMax` (a voice actor of The Simpsons has 30+), then " / …"
87. static rolesMax = 4
88. static rolesText = (&roles) => {
89. let names = []
90. let more = false
91. for (r of listOr(roles)) {
92. let c = r != null ? textOr(r.character) : null
93. if (c != null) {
94. for (part of c.split(' / ')) {
95. let n = part.replaceAll(' (uncredited)', '').trim()
96. if (n != '' && !names.includes(n)) { if (names.length < rolesMax) { names.push(n) } else { more = true } }
97. }
98. }
99. }
100. return names.join(' / ') + (more ? ' / …' : '')
101. }
103. // a series' episodes of one person: TMDB's total_episode_count, else the sum of the roles' / jobs' counts
104. static episodesOf = (&total, &parts) => {
105. if (total != null && hlTypeName(total) == 'Number') { return total }
106. let n = 0
107. for (r of listOr(parts)) { if (r != null && r.episode_count != null && hlTypeName(r.episode_count) == 'Number') { n = n + r.episode_count } }
108. return n
109. }
110. static personOk = (&c) => { return c != null && c.id != null && hlTypeName(c.id) == 'Number' && textOr(c.name) != null && c.adult != true }
112. // TMDB's cast → [{ tmdbId, name, character, gender, episodes }], TMDB's order, at most castMax, adult people left out
113. static castOf = (&d, &movie) => {
114. let src = movie ? (d.credits != null ? d.credits.cast : null) : (d.aggregate_credits != null ? d.aggregate_credits.cast : null)
115. let out = []
116. for (c of listOr(src)) {
117. if (out.length < castMax && personOk(c)) {
118. let role = movie ? textOr(c.character) : rolesText(c.roles)
119. out.push({ tmdbId = c.id name = c.name character = role != null ? role : '' gender = c.gender profile = textOr(c.profile_path) episodes = movie ? null : episodesOf(c.total_episode_count, c.roles) })
120. }
121. }
122. return out
123. }
125. // a series' main cast is in at least this many episodes: half of the series' (rounded up), at most 10 (a season), at least 1
126. static mainMinOf = (&d) => {
127. let n = d.number_of_episodes != null && hlTypeName(d.number_of_episodes) == 'Number' ? d.number_of_episodes : 0
128. let half = (n - n % 2) / 2 + n % 2
129. if (half > 10) { return 10 }
130. return half < 1 ? 1 : half
131. }
133. // the crew jobs we show: TMDB job → ours (the page's labels: components/show.hl crewLabels)
134. static crewJobOf = (&job) => {
135. if (job == 'Director') { return 'director' }
136. if (job == 'Writer') { return 'writer' }
137. if (job == 'Screenplay') { return 'screenplay' }
138. if (job == 'Story') { return 'story' }
139. if (job == 'Original Music Composer' || job == 'Composer' || job == 'Music') { return 'composer' }
140. return null
141. }
142. static crewJobs = ['creator', 'director', 'writer', 'screenplay', 'story', 'composer']
144. // keeps a job's list ordered by episodes (most first, equal ones in TMDB's order), at most crewMax (hand-written: no sort())
145. static insertByEpisodes = (&list, &item) => {
146. let out = []
147. let inserted = false
148. for (x of list) {
149. if (!inserted && item.episodes > x.episodes) { out.push(item) inserted = true }
150. if (out.length < crewMax) { out.push(x) }
151. }
152. if (!inserted && out.length < crewMax) { out.push(item) }
153. return out
154. }
156. // TMDB's crew (+ a series' created_by) → [{ tmdbId, name, job, gender, episodes }], grouped by job in `crewJobs` order, one
157. // entry per person and job
158. static crewOf = (&d, &movie) => {
159. let byJob = {}
160. let seen = {}
161. for (j of crewJobs) { byJob[j] = [] }
162. if (!movie) {
163. for (c of listOr(d.created_by)) {
164. if (personOk(c) && seen['creator:' + c.id] != true && byJob['creator'].length < crewMax) {
165. seen['creator:' + c.id] = true
166. let l = byJob['creator']
167. l.push({ tmdbId = c.id name = c.name job = 'creator' gender = c.gender episodes = 0 })
168. byJob['creator'] = l
169. }
170. }
171. }
172. let src = movie ? (d.credits != null ? d.credits.crew : null) : (d.aggregate_credits != null ? d.aggregate_credits.crew : null)
173. for (c of listOr(src)) {
174. if (personOk(c)) {
175. // a movie's crew entry has one `job`; a series' has `jobs` [{ job, episode_count }]
176. let jobs = movie ? [{ job = c.job episode_count = 0 }] : listOr(c.jobs)
177. for (jb of jobs) {
178. let ours = jb != null ? crewJobOf(jb.job) : null
179. if (ours != null && seen[ours + ':' + c.id] != true) {
180. seen[ours + ':' + c.id] = true
181. let n = movie ? 0 : episodesOf(jb.episode_count, null)
182. byJob[ours] = insertByEpisodes(byJob[ours], { tmdbId = c.id name = c.name job = ours gender = c.gender episodes = n })
183. }
184. }
185. }
186. }
187. let out = []
188. for (j of crewJobs) { for (c of byJob[j]) { out.push(c) } }
189. return out
190. }
192. // the title's cast: TMDB's (linked to our people, new ones added) first, then the entries we had that TMDB does not list.
193. // `mainMin` = a series' main-cast episode count (mainMinOf), 0 for a movie (no guests)
194. static mergedCast = (&show, &tmdbCast, &mainMin) => {
195. let out = []
196. let have = {}
197. let showKey = show.id
198. let seenTmdb = {}
199. for (c of tmdbCast) {
200. let guest = c.episodes != null && c.episodes < mainMin
201. // a guest: linked only when we have them, never created here (mission 062)
202. let pid = guest ? personIdByTmdb(c.tmdbId) : castPersonId(c, showKey)
203. if (pid != null && have[pid] != true) {
204. have[pid] = true
205. seenTmdb['' + c.tmdbId] = true
206. out.push({ person = pid tmdbId = c.tmdbId character = c.character actor = c.name profile = c.profile episodes = c.episodes guest = guest })
207. } else if (pid == null && guest && seenTmdb['' + c.tmdbId] != true) {
208. seenTmdb['' + c.tmdbId] = true
209. out.push({ person = null tmdbId = c.tmdbId character = c.character actor = c.name profile = c.profile episodes = c.episodes guest = true })
210. }
211. }
212. for (c of listOr(show.cast)) {
213. if (c != null && c.person != null && have[c.person] != true && (c.tmdbId == null || seenTmdb['' + c.tmdbId] != true)) {
214. have[c.person] = true
215. out.push(c)
216. }
217. }
218. return out
219. }
221. static crewRoleOf = (&job) => {
222. if (job == 'creator') { return 'Creator' }
223. if (job == 'director') { return 'Director' }
224. if (job == 'writer') { return 'Writer' }
225. if (job == 'screenplay') { return 'Screenplay' }
226. if (job == 'story') { return 'Story' }
227. return 'Music'
228. }
229. // the title's crew: TMDB's, linked to our people (a crew person new to us gets the job as their role) — replaces what we had
230. static linkedCrew = (&show, &tmdbCrew) => {
231. let out = []
232. let showKey = show.id
233. for (c of tmdbCrew) {
234. let pid = castPersonId({ tmdbId = c.tmdbId name = c.name character = crewRoleOf(c.job) gender = c.gender }, showKey)
235. if (pid != null) { out.push({ person = pid name = c.name job = c.job episodes = c.episodes }) }
236. }
237. return out
238. }
240. // the answer carries credits (a details request with `credits` / `aggregate_credits` appended)
241. static hasCredits = (&d, &movie) => { return d != null && (movie ? d.credits != null : d.aggregate_credits != null) }
243. // cast + crew of this record from a details answer → { show (the record with cast, crew, creditsAt), cast, crew, people (new) }
244. static withCredits = (show, &d, &movie) => {
245. let before = castPeopleCount()
246. let tmdbCast = castOf(d, movie)
247. let x = show
248. x.cast = mergedCast(show, tmdbCast, movie ? 0 : mainMinOf(d))
249. x.crew = linkedCrew(show, crewOf(d, movie))
250. x.creditsAt = now()
251. return { show = x cast = tmdbCast.length crew = x.crew.length people = castPeopleCount() - before }
252. }
254. // THE RECORD'S OWN FIELDS from a details answer (tracker#26 completion; tracker#18 the daily sync too — creator: "some shows changed
255. // descriptions and titles and that did not update"): title (a series' `name`, a movie's `title`), TMDB's overview → `tmdbSummary`,
256. // homepage, tagline, status, genres (by TMDB id), adult; type / release / year / language only when we have none. A TMDB value
257. // that is empty never overwrites ours. `summary` is the creator's OWN text — never written here (the page shows summary >
258. // tmdbSummary > tvmazeSummary, shows.hl summaryOf). The slug (`urlSegment`) never changes with the title.
259. // mission 070 (conductor): the sync never DESTROYS what the old tracker's migration brought — before TMDB's value replaces a
260. // migrated record's title / genres / homepage / tagline for the FIRST time, the old one is kept in `migratedTitle` /
261. // `migratedGenres` / `migratedHomepage` / `migratedTagline` (kept, shown nowhere; set once, a later rename leaves it). A record is
262. // migrated when its `oldId` is the old tracker's (not `tmdb-…`: search import, filmography fill). `status`, counts, ids and the
263. // poster are TMDB's state, written by the sync since #9 — not kept.
264. static isMigratedRecord = (&show) => { return show.oldId != null && ('' + show.oldId).slice(0, 5) != 'tmdb-' }
265. static sameIdList = (&a, &b) => {
266. let x = listOr(a)
267. let y = listOr(b)
268. if (x.length != y.length) { return false }
269. let i = 0
270. while (i < x.length) { if ('' + x[i] != '' + y[i]) { return false } i = i + 1 }
271. return true
272. }
273. static withDetailsFields = (show, &d, &movie) => {
274. let x = show
275. let migrated = isMigratedRecord(show)
276. let adult = adultOf(d)
277. if (adult != null) { x.adult = adult }
278. let title = textOr(movie ? d.title : d.name)
279. if (title != null) {
280. if (migrated && x.migratedTitle == null && textOr(x.title) != null && x.title != title) { x.migratedTitle = x.title }
281. x.title = title
282. }
283. if (textOr(d.overview) != null) { x.tmdbSummary = d.overview }
284. if (textOr(d.homepage) != null) {
285. if (migrated && x.migratedHomepage == null && textOr(x.homepage) != null && x.homepage != d.homepage) { x.migratedHomepage = x.homepage }
286. x.homepage = d.homepage
287. }
288. if (textOr(d.tagline) != null) {
289. if (migrated && x.migratedTagline == null && textOr(x.tagline) != null && x.tagline != d.tagline) { x.migratedTagline = x.tagline }
290. x.tagline = d.tagline
291. }
292. if (textOr(d.status) != null) { x.status = d.status }
293. let tmdbGenreIds = []
294. for (g of listOr(d.genres)) { if (g != null && g.id != null) { tmdbGenreIds.push(g.id) } }
295. let genres = genreIdsOfTmdb(tmdbGenreIds)
296. if (genres.length > 0) {
297. if (migrated && x.migratedGenres == null && listOr(x.genres).length > 0 && !sameIdList(x.genres, genres)) { x.migratedGenres = x.genres }
298. x.genres = genres
299. }
300. if (textOr(x.type) == null) { x.type = movie ? 'movie' : 'series' }
301. let release = textOr(movie ? d.release_date : d.first_air_date)
302. if (textOr(x.release) == null && release != null) {
303. x.release = release
304. if (release.length >= 4) { x.year = toNumber(release.slice(0, 4)) }
305. }
306. if (textOr(x.language) == null && textOr(d.original_language) != null) { x.language = d.original_language }
307. // tracker#21 (merge, mission 070): TMDB's TV type + the kind from the genres just stored (the delta's light step and the
308. // completion of #26 store it too, not only syncShow / the kind backfill)
309. if (!movie && textOr(d.type) != null) {
310. x.tmdbType = d.type
311. x.kind = kindOfTitle(d.type, x.genres)
312. }
313. return x
314. }
316. // THE DAILY SYNC (project.hl syncTick, tools/sync-tmdb.hl; tracker#18): after tmdbsync.hl `syncShow` (seasons, episodes, poster,
317. // ids, TVmaze) the record's own fields (withDetailsFields) and its cast + crew (#28) from the SAME details answer (`res.details`)
318. // — no extra request. Answers { renamed (the title changed: the search re-indexes it), cast, crew, people, ms, error }
319. static applySynced = (&showId, &d) => {
320. let out = { renamed = false cast = 0 crew = 0 people = 0 ms = 0 error = '' }
321. let t0 = now()
322. let show = showsTable.fetch(showId)
323. if (show == null || d == null) { return out }
324. let movie = show.type == 'movie'
325. let before = show.title
326. let x = withDetailsFields(show, d, movie)
327. out.renamed = x.title != before
328. if (x.adult != true && hasCredits(d, movie)) {
329. let w = withCredits(x, d, movie)
330. x = w.show
331. out.cast = w.cast
332. out.crew = w.crew
333. out.people = w.people
334. }
335. if (showsTable.update(x.id, x) == null) { out.error = x.title + ': record not stored: ' + showsTable.lastError() }
336. out.ms = now() - t0
337. return out
338. }
340. // the search import: its details answer (tmdbsync.hl `res.details`) → the title's cast + crew. Nothing
341. // for an adult title or an answer without credits. Answers { cast, crew, people, ms, error }
342. static applyCredits = (&showId, &d) => {
343. let out = { cast = 0 crew = 0 people = 0 ms = 0 error = '' }
344. let t0 = now()
345. let show = showsTable.fetch(showId)
346. if (show == null || show.adult == true || adultOf(d) == true || !hasCredits(d, show.type == 'movie')) { return out }
347. let w = withCredits(show, d, show.type == 'movie')
348. if (showsTable.update(show.id, w.show) == null) { out.error = show.title + ': credits not stored: ' + showsTable.lastError() return out }
349. out.cast = w.cast
350. out.crew = w.crew
351. out.people = w.people
352. out.ms = now() - t0
353. return out
354. }
356. // A SERIES IS DONE IN STEPS (real data: The Late Show, 22 seasons / 4,252 episodes, took 10.7 s as one step — hl:web serves one
357. // request at a time, every page waited): step 1 = the details request (fields + cast); then one step per PART of its seasons
358. // (≤ `partEpisodes` episodes, ≤ 20 seasons = one TMDB request — tmdbsync.hl syncShowPart); the last step = ids, TVmaze, poster,
359. // `detailsAt`. A movie is one step. Between two steps every other request is served.
360. // show id → { d (TMDB's answer of step 1), parts ([[season numbers]]), pos, sum } while a series is under way — in memory: a
361. // restart begins again with the details request; what is stored by then is matched by tmdbId, nothing is duplicated
362. static jobs = {}
363. // (TRACKER_DETAILS_PART_EPISODES: the gate sets 1 → one season per step)
364. static partEpisodes = env('TRACKER_DETAILS_PART_EPISODES') != null ? toNumber(env('TRACKER_DETAILS_PART_EPISODES')) : 300
366. static emptySum = () => { return { steps = 0 requests = 0 tvmazeRequests = 0 cast = 0 crew = 0 people = 0 newSeasons = 0 newEpisodes = 0 posters = 0 ids = 0 ms = 0 error = '' } }
367. static addSum = (sum, &r, &ms) => {
368. let o = sum
369. o.steps = o.steps + 1
370. o.requests = o.requests + r.requests
371. o.tvmazeRequests = o.tvmazeRequests + r.tvmazeRequests
372. o.newSeasons = o.newSeasons + r.newSeasons + r.tvmazeSeasons
373. o.newEpisodes = o.newEpisodes + r.newEpisodes + r.tvmazeEpisodes
374. o.posters = o.posters + r.posters
375. o.ids = o.ids + r.ids
376. o.ms = o.ms + ms
377. if (r.error != '') { o.error = r.error }
378. return o
379. }
381. // TMDB's season list → parts of ≤ partEpisodes episodes (a bigger season is a part of its own), ≤ 20 seasons each; which of
382. // them are really synced (season 0, empty ones) syncShowPart decides
383. static seasonParts = (&d) => {
384. let parts = []
385. let cur = []
386. let n = 0
387. for (x of listOr(d.seasons)) {
388. if (x != null && x.season_number != null) {
389. let c = x.episode_count != null ? x.episode_count : 0
390. if (cur.length > 0 && (n + c > partEpisodes || cur.length == 20)) {
391. parts.push(cur)
392. cur = []
393. n = 0
394. }
395. cur.push(x.season_number)
396. n = n + c
397. }
398. }
399. if (cur.length > 0) { parts.push(cur) }
400. return parts
401. }
403. // a step's answer: { skipped (nothing to do), done, requests / tvmazeRequests (THIS step's — what the pause counts), adult,
404. // error ('' = none), retry (a passing failure: no answer, 429, 5xx), sum (the whole title's counts so far: steps, requests,
405. // cast, newSeasons, newEpisodes, posters, ids, ms) }
406. static stepOut = () => { return { skipped = false done = false requests = 0 tvmazeRequests = 0 adult = null error = '' retry = false sum = emptySum() } }
408. // the last step: the title is complete
409. static markComplete = (&showId, &sum) => {
410. let done = showsTable.fetch(showId)
411. done.detailsAt = now()
412. if (sum.error != '') { done.detailsError = sum.error }
413. let ok = showsTable.update(done.id, done) != null
414. refreshCatalogShow(done.id)
415. searchTitleChanged(done.id)
416. return ok
417. }
419. // step 1: the details request — the record's fields, the cast; a movie also its ids + poster (done); a series' parts are planned
420. static firstStep = (show) => {
421. let out = stepOut()
422. let t0 = now()
423. let movie = show.type == 'movie'
424. out.requests = 1
425. let d = tmdbGet((movie ? '/movie/' : '/tv/') + show.tmdbId + '?append_to_response=external_ids,' + (movie ? 'credits' : 'aggregate_credits'))
426. if (d.failed != null) {
427. out.error = show.title + ': ' + d.failed
428. if (d.status == 404) {
429. show.detailsCheck = 'TMDB 404'
430. showsTable.update(show.id, show)
431. out.done = true
432. } else { out.retry = true }
433. return out
434. }
435. out.adult = adultOf(d)
436. let x = withDetailsFields(show, d, movie)
437. // mission 025: a movie has no season/episode counts (hasTvLeftovers)
438. if (movie) { x.seasonsCount = null x.episodesCount = null }
439. if (out.adult == true) {
440. // never public: the flag, no poster, no seasons, no cast (mission 056)
441. x.detailsCheck = 'adult'
442. showsTable.update(x.id, x)
443. refreshCatalogShow(x.id)
444. searchTitleChanged(x.id)
445. out.done = true
446. return out
447. }
448. // tracker#28: the whole cast + the crew, from this same answer
449. let w = withCredits(x, d, movie)
450. x = w.show
451. if (showsTable.update(x.id, x) == null) {
452. out.error = show.title + ': not updated: ' + showsTable.lastError()
453. out.retry = true
454. return out
455. }
456. let sum = emptySum()
457. sum.cast = w.cast
458. sum.crew = w.crew
459. sum.people = w.people
460. if (movie) {
461. // one request was enough: ids + poster now (syncMovie with this answer)
462. let r = syncShowWith(x.id, d)
463. out.requests = r.requests
464. out.sum = addSum(sum, r, now() - t0)
465. if (r.error != '') { out.error = r.error }
466. markComplete(x.id, out.sum)
467. out.done = true
468. return out
469. }
470. sum.steps = 1
471. sum.requests = 1
472. sum.ms = now() - t0
473. // the next steps need the details, not the credits (a long series' aggregate_credits are ~1 MB, copied on every step)
474. let dd = d
475. dd.aggregate_credits = null
476. jobs[x.id] = { d = dd parts = seasonParts(d) pos = 0 sum = sum }
477. out.sum = sum
478. return out
479. }
481. // ONE step of completing this title (by our id) — called until `done` (the page's face, the repair job)
482. static completeStep = (&showId) => {
483. let show = showsTable.fetch(showId)
484. if (!needsDetails(show)) {
485. jobs[showId] = null
486. let out = stepOut()
487. out.skipped = true
488. out.done = true
489. return out
490. }
491. let job = jobs[showId]
492. if (job == null) { return firstStep(show) }
493. let t0 = now()
494. let out = stepOut()
495. let last = job.pos >= job.parts.length
496. let r = syncShowPart(showId, job.d, { seasons = last ? [] : job.parts[job.pos] finish = last })
497. out.requests = r.requests
498. out.tvmazeRequests = r.tvmazeRequests
499. if (r.error != '') { out.error = r.error }
500. let j = job
501. j.sum = addSum(j.sum, r, now() - t0)
502. j.pos = j.pos + 1
503. out.sum = j.sum
504. if (!last) {
505. jobs[showId] = j
506. return out
507. }
508. jobs[showId] = null
509. markComplete(showId, j.sum)
510. out.done = true
511. return out
512. }
514. // the line every completed title logs
515. static completeLine = (&title, &sum) => {
516. return '"' + title + '" steps=' + sum.steps + ' requests=' + sum.requests + ' tvmazeRequests=' + sum.tvmazeRequests + ' cast=' + sum.cast + ' newSeasons=' + sum.newSeasons
517. + ' newEpisodes=' + sum.newEpisodes + ' posters=' + sum.posters + ' ids=' + sum.ids + ' ms=' + sum.ms + ' crew=' + sum.crew + ' people=' + sum.people + (sum.error != '' ? ' error: ' + sum.error : '')
518. }
520. // flush shows/seasons/episodes + people (new cast members)
521. static persistDetails = () => {
522. persistSync()
523. personsTable.persist()
524. return true
525. }
526. // the credits job touches only shows + people: the episodes' index files (~16 MB on the live data) are not rewritten
527. static persistCredits = () => {
528. showsTable.persist()
529. personsTable.persist()
530. return true
531. }
533. // ---- THE CREDITS JOB (tracker#28, project.hl `creditsTick`): every public title that has no credits yet, once --------------
534. // (the incomplete ones get theirs from completeStep; adult and unknown-flag titles are never asked — not public)
535. static needsCredits = (&s) => { return s != null && s.tmdbId != null && s.adult == false && s.creditsAt == null && s.creditsCheck == null && !isIncomplete(s) && s.mergedInto == null }
536. // in table order, read once when the job starts (a restart goes on with what is left)
537. static creditsQueue = () => {
538. let out = []
539. for (s of showsTable.find(null, null)) { if (needsCredits(s)) { out.push(s.id) } }
540. return out
541. }
542. // ONE title: one request (details + credits / aggregate_credits) → its cast + crew. Answers { skipped, requests, cast, crew,
543. // people, ms, error, retry (no answer, 429, 5xx: ask again later) }. TMDB 404 / adult → `creditsCheck`, never asked again.
544. static creditsStep = (&showId) => {
545. let out = { skipped = false requests = 0 cast = 0 crew = 0 people = 0 ms = 0 error = '' retry = false }
546. let t0 = now()
547. let show = showsTable.fetch(showId)
548. if (!needsCredits(show)) { out.skipped = true return out }
549. let movie = show.type == 'movie'
550. out.requests = 1
551. let d = tmdbGet((movie ? '/movie/' : '/tv/') + show.tmdbId + '?append_to_response=' + (movie ? 'credits' : 'aggregate_credits'))
552. if (d.failed != null) {
553. out.error = show.title + ': ' + d.failed
554. if (d.status == 404) {
555. show.creditsCheck = 'TMDB 404'
556. showsTable.update(show.id, show)
557. } else { out.retry = true }
558. return out
559. }
560. if (adultOf(d) == true) {
561. show.creditsCheck = 'adult'
562. showsTable.update(show.id, show)
563. return out
564. }
565. let w = withCredits(show, d, movie)
566. if (showsTable.update(show.id, w.show) == null) {
567. out.error = show.title + ': credits not stored: ' + showsTable.lastError()
568. out.retry = true
569. return out
570. }
571. out.cast = w.cast
572. out.crew = w.crew
573. out.people = w.people
574. out.ms = now() - t0
575. return out
576. }