tracker

29.5 KB

1. // lib/merge.hl — DUPLICATE TITLES MERGED (tracker.worldapi.org#31, mission 026). The architect: "Every TMDB title exists once in our
2. // data; watches, follows and seasons all hang on that one record."
3. //
4. // WHERE THEY CAME FROM: the old tracker. On the live copy (2026-10-03) 68 type+tmdbId pairs were held by 157 records — every one
5. // of them a migrated row (Mongo `oldId`, created 2022–2023 in runs of consecutive ids: Archive 81 ×6, Peripheral ×4, Ally
6. // McBeal ×2 …); not one was made by this app. The app's two creating paths look the title up first (search.hl importTitle,
7. // people.hl fillStep → search.hl titleIdByTmdb) and their records carry `oldId = tmdb-<tv|movie>-<id>`, which the table's
8. // `!oldId` UNIQUE index refuses twice. Seasons the same way: 449 migrated seasons name a title that does not list them (the old
9. // tracker showed a title's seasons by query); the daily sync goes by the list, so it made a SECOND season where TMDB has that
10. // number (Reacher: the creator's S3 watches sat on the unlisted migrated S3, the page showed the sync's empty S3).
11. //
12. // THE REPAIR (in the app, on its own clock like the short-id backfill — no TMDB, one item per tick; jobs.hl `mergeTick`):
13. // 1. per duplicate group: the KEEPER = the record with follows/watches (most first), else the one with its old short id
14. // (shortids.hl), else the oldest (the Mongo id's time, else `imported`). Into it, from every other record:
15. // follows + watches (a user who already has the keeper's: the moved row gets `target`/`show` null + `mergedInto` = the
16. // row it repeats — no row is deleted), seasons the keeper lacks (moved: `show` = keeper), seasons it has (merged
17. // episode by episode, below), cast + crew entries it lacks, the persons' credits, the timelines (franchises.hl
18. // titleMerged), empty text/id fields, the poster file if the keeper has none. The keeper takes the plain slug of its
19. // title when a duplicate held it (Archive 81: "Archive-82" → "Archive-81"; its former slug in `oldSlugs`).
20. // The other record becomes a TOMBSTONE (shows.hl): `mergedInto`, `mergedAt`, `mergedTmdbId` (tmdbId null), `mergedSeasons`
21. // (seasons []). Its slug and short id lead to the keeper (shows.hl movedSlugs → project.hl 301, shortids.hl shortIdPath).
22. // 2. per title with seasons that name it but are not in its list (STRAY seasons): such a season with a listed TWIN (same
23. // number) is merged into the twin; one without is LINKED into the list only when someone watched it or one of its episodes
24. // (recorded in the title's `linkedSeasons`). The other strays stay as they are: linking all 393 with episodes on the live
25. // copy put 1,091 never-watched old talk-show episodes on the creator's /my/unwatched (1,273 → 2,364 rows). When the sync
26. // later makes a season of that number, the next start merges the stray into it.
27. // MERGING A SEASON INTO ITS TWIN: the season's watches → the twin; each episode: the twin's episode of that number gets its
28. // watches (the episode becomes `mergedInto` = that one), an episode the twin lacks MOVES into the twin. The season gets
29. // `mergedInto` = the twin, `mergedEpisodes` (episodes []).
30. // 3. tracker#39 (mission 032): per listed season with two or more live EPISODES of one number (DOUBLE EPISODES: a migrated
31. // episode whose TMDB id TMDB had replaced + the one the sync then added — sync.hl matches those by number now; live copy
32. // 2026-10-04: 850 in 112 series): the KEEPER = the one with the most watches, else the synced one (`tmdb-episode:` oldId:
33. // TMDB's current id), else the first listed. It gets the others' watches; a migrated keeper takes the synced one's tmdbId
34. // (its own → `migratedTmdbId`, so the next sync matches it) and TMDB's texts (its own kept in migratedTitle/migratedSummary),
35. // empty fields filled. The others become tombstones: `mergedInto`, `mergedAt`, `mergedTmdbId` (tmdbId null), out of the
36. // season's `episodes` into its `mergedEpisodes`. Nothing is deleted.
37. // RESUMES: the queue is built at start from what is still duplicated / stray — a restart in the middle goes on (a half-moved
38. // group picks the same keeper again: it already has the moved follows/watches). A third start: "merge: nothing to do".
39. // Persisted every 10 items and at the end. Log: start, one line per tombstone and per repaired title, done.
40. //
41. // ONE PROCESS PER STORAGE (hybriel#21): the tables are the app's own. Ids are unique (count-duplicate-ids), so update() is safe
42. // (hybriel#113, sync.hl).
43. import { now } from 'hl:time'
44. import { readBytes, writeFile, exists } from 'hl:fs'
45. import { postersDir, hexDigits, listOr } from './util.hl'
46. import { showsTable, seasonsTable, episodesTable, personsTable, noteSlugsOf, forgetEpisodeKey, posterName } from './shows.hl'
47. import { followsTable } from './follows.hl'
48. import { watchesTable, forgetUserWatches } from './watches.hl'
49. import { searchTitleMerged } from './search.hl'
50. import { slugBaseOf } from './search-helpers.hl'
51. import { refreshCatalogShow } from './catalog.hl'
52. import { hasOldShortId } from './shortids.hl'
53. import { titleMerged } from './franchises.hl'
55. // the empty fields of a keeper a duplicate may fill (never ids, slugs, flags or the job markers)
56. static fillFields = ['summary', 'tmdbSummary', 'tvmazeSummary', 'imdbId', 'tvdbId', 'tvmzId', 'homepage', 'tagline', 'release', 'year', 'language',
57. 'migratedSummary', 'migratedTitle', 'migratedGenres', 'migratedHomepage', 'migratedTagline']
59. // the work, [{ kind ('title' | 'seasons'), ids (a group's record ids) | id (a title) }], worked off from `run.pos`; filled in place
60. // (a static list is never reassigned)
61. static queue = []
62. // watch id lists by target, follow id lists by show, person ids by the duplicates' ids they credit, seasons naming a title that
63. // does not list them — built once by mergePrepare, kept up to date by the moves (flat maps, written key by key)
64. static watchIds = {}
65. static followIds = {}
66. static creditedBy = {}
67. static strays = {}
68. static seasonsByDup = {}
69. static run = { built = false pos = 0 groups = 0 records = 0 strayTitles = 0 doubleSeasons = 0 ms = 0 }
70. static totals = { groups = 0 tombstones = 0 follows = 0 followRepeats = 0 watches = 0 watchRepeats = 0 seasonsMoved = 0 seasonsMerged = 0
71. seasonsLinked = 0 episodesMoved = 0 episodesMerged = 0 cast = 0 people = 0 timelines = 0 slugs = 0 posters = 0 fields = 0 titles = 0
72. doubleSeasons = 0 doubleEpisodes = 0 }
74. static isEmpty = (&v) => { return v == null || (hlTypeName(v) == 'String' && v.trim() == '') }
76. // ---- the queue ------------------------------------------------------------------------------------------------------------
77. // one scan of shows, seasons, follows, watches and persons: the duplicate groups, the titles with stray seasons, the indexes
78. static mergePrepare = () => {
79. if (run.built) { return queue.length }
80. run.built = true
81. t0 = now()
82. byKey = {}
83. order = []
84. listedBy = {}
85. for (s of showsTable.find(null, null)) {
86. if (s.mergedInto == null) {
87. for (x of listOr(s.seasons)) { listedBy[x] = s.id }
88. if (s.tmdbId != null) {
89. let k = s.type + ':' + s.tmdbId
90. if (byKey[k] == null) {
91. byKey[k] = [s.id]
92. order.push(k)
93. } else { byKey[k].push(s.id) }
94. }
95. }
96. }
97. dup = {}
98. for (k of order) {
99. let ids = byKey[k]
100. if (ids.length > 1) {
101. queue.push({ kind = 'title' key = k ids = ids })
102. run.groups = run.groups + 1
103. run.records = run.records + ids.length
104. for (id of ids) { dup[id] = true }
105. }
106. }
107. for (w of watchesTable.find(null, null)) {
108. if (w.target != null) { if (watchIds[w.target] == null) { watchIds[w.target] = [w.id] } else { watchIds[w.target].push(w.id) } }
109. }
110. // seasons: a duplicate's (all of them, listed or not) and the stray ones (naming a live title that does not list them)
111. listedNum = {}
112. strayList = []
113. for (x of seasonsTable.find(null, null)) {
114. if (x.show != null && x.mergedInto == null) {
115. if (dup[x.show] == true) {
116. if (seasonsByDup[x.show] == null) { seasonsByDup[x.show] = [x.id] } else { seasonsByDup[x.show].push(x.id) }
117. }
118. if (listedBy[x.id] != null) { listedNum[listedBy[x.id] + '#' + x.seasonNumber] = true }
119. else { strayList.push({ id = x.id show = x.show n = x.seasonNumber watched = hasWatches(x) }) }
120. }
121. }
122. // a stray season is work when it has a listed twin or watches; its title must be live and no duplicate (a duplicate's seasons
123. // go with its group)
124. strayShows = []
125. for (st of strayList) {
126. if (dup[st.show] != true && (st.watched || listedNum[st.show + '#' + st.n] == true)) {
127. if (strays[st.show] == null) {
128. strays[st.show] = [st.id]
129. strayShows.push(st.show)
130. } else { strays[st.show].push(st.id) }
131. }
132. }
133. for (id of strayShows) {
134. let s = showsTable.fetch(id)
135. if (s != null && s.mergedInto == null) {
136. queue.push({ kind = 'seasons' id = id })
137. run.strayTitles = run.strayTitles + 1
138. }
139. }
140. // tracker#39: the listed seasons holding two live episodes of one number (one pass over the episodes, one at a time)
141. firstOfNumber = {}
142. doubled = {}
143. for (e of episodesTable.iterate(null, null)) {
144. if (e.mergedInto == null && e.season != null && listedBy[e.season] != null) {
145. let k = e.season + '#' + e.episodeNumber
146. if (firstOfNumber[k] == null) { firstOfNumber[k] = true }
147. else if (doubled[e.season] == null) {
148. doubled[e.season] = true
149. queue.push({ kind = 'episodes' id = e.season show = listedBy[e.season] })
150. run.doubleSeasons = run.doubleSeasons + 1
151. }
152. }
153. }
154. if (queue.length > 0) {
155. for (f of followsTable.find(null, null)) {
156. if (f.show != null) { if (followIds[f.show] == null) { followIds[f.show] = [f.id] } else { followIds[f.show].push(f.id) } }
157. }
158. if (run.groups > 0) {
159. for (p of personsTable.find(null, null)) {
160. if (p.shows != null) {
161. for (c of p.shows) {
162. if (c != null && c.show != null && dup[c.show] == true) {
163. if (creditedBy[c.show] == null) { creditedBy[c.show] = [p.id] } else if (!creditedBy[c.show].includes(p.id)) { creditedBy[c.show].push(p.id) }
164. }
165. }
166. }
167. }
168. }
169. }
170. run.ms = now() - t0
171. return queue.length
172. }
173. // anyone's watch on the season or one of its episodes (watchIds is built)
174. static hasWatches = (&season) => {
175. if (watchIds[season.id] != null && watchIds[season.id].length > 0) { return true }
176. for (eid of listOr(season.episodes)) { if (watchIds[eid] != null && watchIds[eid].length > 0) { return true } }
177. return false
178. }
179. static mergeLeft = () => { return queue.length - run.pos }
180. static mergeStats = () => { return run }
181. static mergeTotals = () => { return totals }
182. static mergeLine = () => {
183. t = totals
184. return 'groups=' + t.groups + ' tombstones=' + t.tombstones + ' follows=' + t.follows + ' followRepeats=' + t.followRepeats + ' watches=' + t.watches
185. + ' watchRepeats=' + t.watchRepeats + ' seasonsMoved=' + t.seasonsMoved + ' seasonsMerged=' + t.seasonsMerged + ' seasonsLinked=' + t.seasonsLinked
186. + ' episodesMoved=' + t.episodesMoved + ' episodesMerged=' + t.episodesMerged + ' cast=' + t.cast + ' people=' + t.people + ' timelines=' + t.timelines
187. + ' slugs=' + t.slugs + ' posters=' + t.posters + ' fields=' + t.fields + ' strayTitles=' + t.titles + ' doubleSeasons=' + t.doubleSeasons
188. + ' doubleEpisodes=' + t.doubleEpisodes
189. }
190. static persistMerge = () => {
191. showsTable.persist()
192. seasonsTable.persist()
193. episodesTable.persist()
194. followsTable.persist()
195. watchesTable.persist()
196. personsTable.persist()
197. return true
198. }
200. // ---- moving user data ------------------------------------------------------------------------------------------------------
201. // every watch of `fromId` → `toId` (an episode, season or movie id); a user who already has a watch on `toId` keeps that one and
202. // the moved row is parked (target null, `mergedInto` = the row it repeats). Every moved row keeps where it was (`mergedTarget`).
203. static moveWatches = (&fromId, &toId) => {
204. if (watchIds[fromId] == null) { return 0 }
205. ids = []
206. for (x of watchIds[fromId]) { ids.push(x) }
207. mine = {}
208. if (watchIds[toId] != null) {
209. for (x of watchIds[toId]) {
210. let w = watchesTable.fetch(x)
211. if (w != null && w.target == toId) { mine[w.user] = w.id }
212. }
213. }
214. moved = []
215. let n = 0
216. for (wid of ids) {
217. let w = watchesTable.fetch(wid)
218. if (w != null && w.target == fromId) {
219. let x = w
220. x.mergedTarget = fromId
221. if (mine[x.user] != null) {
222. x.target = null
223. x.mergedInto = mine[x.user]
224. totals.watchRepeats = totals.watchRepeats + 1
225. } else {
226. x.target = toId
227. mine[x.user] = x.id
228. moved.push(x.id)
229. totals.watches = totals.watches + 1
230. }
231. if (watchesTable.update(x.id, x) == null) { console.log('merge: watch ' + x.id + ' not stored: ' + watchesTable.lastError()) } else { n = n + 1 }
232. forgetUserWatches(x.user)
233. }
234. }
235. watchIds[fromId] = []
236. if (moved.length > 0) {
237. if (watchIds[toId] == null) { watchIds[toId] = moved } else { for (x of moved) { watchIds[toId].push(x) } }
238. }
239. return n
240. }
241. // every follow of title `fromId` → `toId`, the same way (`mergedShow` = where it was)
242. static moveFollows = (&fromId, &toId) => {
243. if (followIds[fromId] == null) { return 0 }
244. ids = []
245. for (x of followIds[fromId]) { ids.push(x) }
246. mine = {}
247. if (followIds[toId] != null) {
248. for (x of followIds[toId]) {
249. let f = followsTable.fetch(x)
250. if (f != null && f.show == toId) { mine[f.user] = f.id }
251. }
252. }
253. moved = []
254. for (fid of ids) {
255. let f = followsTable.fetch(fid)
256. if (f != null && f.show == fromId) {
257. let x = f
258. x.mergedShow = fromId
259. if (mine[x.user] != null) {
260. x.show = null
261. x.mergedInto = mine[x.user]
262. totals.followRepeats = totals.followRepeats + 1
263. } else {
264. x.show = toId
265. mine[x.user] = x.id
266. moved.push(x.id)
267. totals.follows = totals.follows + 1
268. }
269. if (followsTable.update(x.id, x) == null) { console.log('merge: follow ' + x.id + ' not stored: ' + followsTable.lastError()) }
270. }
271. }
272. followIds[fromId] = []
273. if (moved.length > 0) {
274. if (followIds[toId] == null) { followIds[toId] = moved } else { for (x of moved) { followIds[toId].push(x) } }
275. }
276. return moved.length
277. }
279. // ---- seasons -------------------------------------------------------------------------------------------------------------------
280. // season `x` into its twin `twinId` of title `titleId` (see the top) → the episode ids the twin gained
281. static mergeSeasonInto = (&seasonId, &twinId, &titleId) => {
282. x = seasonsTable.fetch(seasonId)
283. twin = seasonsTable.fetch(twinId)
284. if (x == null || twin == null || x.mergedInto != null) { return [] }
285. moveWatches(x.id, twin.id)
286. byNum = {}
287. for (eid of listOr(twin.episodes)) {
288. let e = episodesTable.fetch(eid)
289. if (e != null) { byNum['' + e.episodeNumber] = e.id }
290. }
291. gained = []
292. tid = twin.id
293. for (eid of listOr(x.episodes)) {
294. let e = episodesTable.fetch(eid)
295. if (e != null && e.mergedInto == null) {
296. let other = byNum['' + e.episodeNumber]
297. let y = e
298. if (other != null && other != e.id) {
299. moveWatches(e.id, other)
300. y.mergedInto = other
301. totals.episodesMerged = totals.episodesMerged + 1
302. } else if (other == null) {
303. y.movedFrom = { show = e.show season = e.season }
304. y.show = titleId
305. y.season = tid
306. byNum['' + e.episodeNumber] = e.id
307. gained.push(e.id)
308. totals.episodesMoved = totals.episodesMoved + 1
309. }
310. if (episodesTable.update(y.id, y) == null) { console.log('merge: episode ' + y.id + ' not stored: ' + episodesTable.lastError()) }
311. forgetEpisodeKey(y.id)
312. }
313. }
314. if (gained.length > 0) {
315. tw = seasonsTable.fetch(tid)
316. eps = listOr(tw.episodes)
317. for (g of gained) { eps.push(g) }
318. tw.episodes = eps
319. if (seasonsTable.update(tw.id, tw) == null) { console.log('merge: season ' + tw.id + ' not stored: ' + seasonsTable.lastError()) }
320. }
321. done = seasonsTable.fetch(seasonId)
322. done.mergedInto = tid
323. done.mergedAt = now()
324. done.mergedEpisodes = listOr(done.episodes)
325. done.episodes = []
326. if (seasonsTable.update(done.id, done) == null) { console.log('merge: season ' + done.id + ' not stored: ' + seasonsTable.lastError()) }
327. totals.seasonsMerged = totals.seasonsMerged + 1
328. return gained
329. }
330. // a season (and its episodes) moved to title `titleId` — it was a duplicate's and the keeper has no season of its number
331. static moveSeason = (&seasonId, &titleId) => {
332. x = seasonsTable.fetch(seasonId)
333. if (x == null) { return false }
334. y = x
335. y.movedFrom = { show = x.show }
336. y.show = titleId
337. if (seasonsTable.update(y.id, y) == null) {
338. console.log('merge: season ' + y.id + ' not stored: ' + seasonsTable.lastError())
339. return false
340. }
341. for (eid of listOr(y.episodes)) {
342. let e = episodesTable.fetch(eid)
343. if (e != null) {
344. let z = e
345. z.movedFrom = { show = e.show season = e.season }
346. z.show = titleId
347. if (episodesTable.update(z.id, z) == null) { console.log('merge: episode ' + z.id + ' not stored: ' + episodesTable.lastError()) }
348. forgetEpisodeKey(z.id)
349. }
350. }
351. totals.seasonsMoved = totals.seasonsMoved + 1
352. return true
353. }
354. // a title's listed seasons by number → season id
355. static seasonsByNumber = (&show) => {
356. out = {}
357. for (sid of listOr(show.seasons)) {
358. let s = seasonsTable.fetch(sid)
359. if (s != null) { out['' + s.seasonNumber] = s.id }
360. }
361. return out
362. }
364. // ---- the keeper ---------------------------------------------------------------------------------------------------------------
365. // follows + watches on the record, its seasons and their episodes
366. static userDataOf = (&r) => {
367. let n = listOr(followIds[r.id]).length + listOr(watchIds[r.id]).length
368. sids = []
369. for (x of listOr(r.seasons)) { sids.push(x) }
370. for (x of listOr(seasonsByDup[r.id])) { if (!sids.includes(x)) { sids.push(x) } }
371. for (sid of sids) {
372. n = n + listOr(watchIds[sid]).length
373. let s = seasonsTable.fetch(sid)
374. if (s != null) { for (eid of listOr(s.episodes)) { n = n + listOr(watchIds[eid]).length } }
375. }
376. return n
377. }
378. // when the record was made (ms): a Mongo id's first 8 hex digits are its seconds; else `imported`; else last
379. static ageOf = (&r) => {
380. o = r.oldId != null ? '' + r.oldId : ''
381. if (o.length == 24) {
382. let v = 0
383. let i = 0
384. let ok = true
385. while (ok && i < 8) {
386. let d = hexDigits.indexOf(o.slice(i, i + 1))
387. if (d < 0) { ok = false } else { v = v * 16 + d }
388. i = i + 1
389. }
390. if (ok) { return v * 1000 }
391. }
392. if (r.imported != null && hlTypeName(r.imported) == 'Number') { return r.imported }
393. return 9000000000000000
394. }
395. // → the keeper's id: most user data, then the old short id, then the oldest; equal → the first
396. static pickKeeper = (&recs) => {
397. let best = null
398. let bestData = -1
399. let bestOld = false
400. let bestAge = 0
401. for (r of recs) {
402. let d = userDataOf(r)
403. let o = hasOldShortId(r)
404. let a = ageOf(r)
405. let better = best == null || d > bestData || (d == bestData && o && !bestOld) || (d == bestData && o == bestOld && a < bestAge)
406. if (better) {
407. best = r.id
408. bestData = d
409. bestOld = o
410. bestAge = a
411. }
412. }
413. return best
414. }
416. // ---- one duplicate into its keeper ---------------------------------------------------------------------------------------------
417. static creditKey = (&c) => { return c.person != null ? 'p' + c.person : 't' + c.tmdbId }
418. static mergeTitleInto = (&dupId, &keeperId) => {
419. t = showsTable.fetch(dupId)
420. k0 = showsTable.fetch(keeperId)
421. if (t == null || k0 == null || t.mergedInto != null) { return null }
422. line = { follows = moveFollows(t.id, keeperId) watches = moveWatches(t.id, keeperId) seasons = 0 merged = 0 people = 0 }
423. // seasons: the listed ones first, then the ones naming it
424. sids = []
425. for (x of listOr(t.seasons)) { sids.push(x) }
426. for (x of listOr(seasonsByDup[t.id])) { if (!sids.includes(x)) { sids.push(x) } }
427. have = seasonsByNumber(k0)
428. added = []
429. for (sid of sids) {
430. let s = seasonsTable.fetch(sid)
431. if (s != null && s.mergedInto == null) {
432. let twin = have['' + s.seasonNumber]
433. if (twin != null && twin != s.id) {
434. mergeSeasonInto(s.id, twin, keeperId)
435. line.merged = line.merged + 1
436. } else if (twin == null && moveSeason(s.id, keeperId)) {
437. have['' + s.seasonNumber] = s.id
438. added.push(s.id)
439. line.seasons = line.seasons + 1
440. }
441. }
442. }
443. // the persons' credits
444. for (pid of listOr(creditedBy[t.id])) {
445. let p = personsTable.fetch(pid)
446. if (p != null) {
447. let shows = []
448. let changed = false
449. for (c of listOr(p.shows)) {
450. let x = c
451. if (x != null && x.show == t.id) {
452. x.show = keeperId
453. changed = true
454. }
455. shows.push(x)
456. }
457. if (changed) {
458. let q = p
459. q.shows = shows
460. if (personsTable.update(q.id, q) == null) { console.log('merge: person ' + q.id + ' not stored: ' + personsTable.lastError()) } else { line.people = line.people + 1 }
461. }
462. }
463. }
464. totals.people = totals.people + line.people
465. totals.timelines = totals.timelines + titleMerged(t.id, keeperId)
467. // the keeper: its new seasons, the cast/crew it lacks, empty fields, the poster, the plain slug
468. k = showsTable.fetch(keeperId)
469. seasons = listOr(k.seasons)
470. for (a of added) { seasons.push(a) }
471. k.seasons = seasons
472. castKeys = {}
473. cast = listOr(k.cast)
474. for (c of cast) { if (c != null) { castKeys[creditKey(c)] = true } }
475. for (c of listOr(t.cast)) {
476. if (c != null && castKeys[creditKey(c)] != true) {
477. castKeys[creditKey(c)] = true
478. cast.push(c)
479. totals.cast = totals.cast + 1
480. }
481. }
482. k.cast = cast
483. crewKeys = {}
484. crew = listOr(k.crew)
485. for (c of crew) { if (c != null) { crewKeys[c.person + '|' + c.name + '|' + c.job] = true } }
486. for (c of listOr(t.crew)) {
487. if (c != null && crewKeys[c.person + '|' + c.name + '|' + c.job] != true) {
488. crewKeys[c.person + '|' + c.name + '|' + c.job] = true
489. crew.push(c)
490. totals.cast = totals.cast + 1
491. }
492. }
493. if (crew.length > 0) { k.crew = crew }
494. for (f of fillFields) {
495. if (isEmpty(k[f]) && !isEmpty(t[f])) {
496. k[f] = t[f]
497. totals.fields = totals.fields + 1
498. }
499. }
500. kp = posterName(k)
501. tp = posterName(t)
502. if ((kp == null || !exists(postersDir + '/' + kp)) && tp != null && exists(postersDir + '/' + tp) && k.oldId != null) {
503. writeFile(postersDir + '/' + k.oldId + '.' + t.image, readBytes(postersDir + '/' + tp), 420)
504. k.image = t.image
505. totals.posters = totals.posters + 1
506. }
507. plain = slugBaseOf(k.title, '')
508. if (plain != '' && t.urlSegment == plain && k.urlSegment != plain) {
509. olds = listOr(k.oldSlugs)
510. if (k.urlSegment != null && !olds.includes(k.urlSegment)) { olds.push(k.urlSegment) }
511. k.oldSlugs = olds
512. k.urlSegment = plain
513. totals.slugs = totals.slugs + 1
514. }
515. if (showsTable.update(k.id, k) == null) { console.log('merge: keeper ' + k.id + ' not stored: ' + showsTable.lastError()) }
517. // the duplicate becomes a tombstone (last: a stop before this leaves a group the next start finishes)
518. d = showsTable.fetch(dupId)
519. tmdbId = d.tmdbId
520. type = d.type
521. d.mergedInto = keeperId
522. d.mergedAt = now()
523. d.mergedTmdbId = tmdbId
524. d.tmdbId = null
525. d.mergedSeasons = listOr(d.seasons)
526. d.seasons = []
527. if (showsTable.update(d.id, d) == null) {
528. console.log('merge: tombstone ' + d.id + ' not stored: ' + showsTable.lastError())
529. return null
530. }
531. totals.tombstones = totals.tombstones + 1
532. noteSlugsOf(d)
533. noteSlugsOf(showsTable.fetch(keeperId))
534. searchTitleMerged(d.id, keeperId, type, tmdbId)
535. refreshCatalogShow(d.id)
536. refreshCatalogShow(keeperId)
537. console.log('merge: "' + d.title + '" ' + d.id + ' [' + d.urlSegment + ' ' + d.shortId + '] → ' + keeperId + ' [' + showsTable.fetch(keeperId).urlSegment + ']'
538. + ' follows=' + line.follows + ' watches=' + line.watches + ' seasonsMoved=' + line.seasons + ' seasonsMerged=' + line.merged + ' people=' + line.people)
539. return line
540. }
542. // ---- one title's stray seasons ----------------------------------------------------------------------------------------------
543. static straySeasonsStep = (&showId) => {
544. show = showsTable.fetch(showId)
545. if (show == null || show.mergedInto != null) { return null }
546. listed = listOr(show.seasons)
547. have = seasonsByNumber(show)
548. // the ones with more episodes first: of two watched strays of one number the fuller one is linked, the other merged into it
549. let rows = []
550. for (sid of listOr(strays[showId])) {
551. let s = seasonsTable.fetch(sid)
552. if (s != null && s.mergedInto == null && s.show == showId && !listed.includes(s.id)) {
553. let row = { id = s.id n = s.seasonNumber eps = listOr(s.episodes).length watched = hasWatches(s) }
554. let out = []
555. let put = false
556. for (r of rows) {
557. if (!put && row.eps > r.eps) { out.push(row) put = true }
558. out.push(r)
559. }
560. if (!put) { out.push(row) }
561. rows = out
562. }
563. }
564. linked = []
565. let merged = 0
566. for (r of rows) {
567. let twin = have['' + r.n]
568. if (twin != null) {
569. mergeSeasonInto(r.id, twin, showId)
570. merged = merged + 1
571. } else if (r.watched) {
572. linked.push(r.id)
573. have['' + r.n] = r.id
574. totals.seasonsLinked = totals.seasonsLinked + 1
575. }
576. }
577. if (linked.length > 0) {
578. x = showsTable.fetch(showId)
579. seasons = listOr(x.seasons)
580. marks = listOr(x.linkedSeasons)
581. for (l of linked) {
582. seasons.push(l)
583. marks.push(l)
584. }
585. x.seasons = seasons
586. // which ones the repair linked (they were in the data, not in the list)
587. x.linkedSeasons = marks
588. if (showsTable.update(x.id, x) == null) { console.log('merge: title ' + x.id + ' not stored: ' + showsTable.lastError()) }
589. }
590. if (linked.length > 0 || merged > 0) {
591. totals.titles = totals.titles + 1
592. refreshCatalogShow(showId)
593. console.log('merge: seasons of "' + show.title + '" ' + showId + ': linked=' + linked.length + ' merged=' + merged)
594. }
595. return null
596. }
598. // ---- double episodes (tracker#39) ----------------------------------------------------------------------------------------------
599. static isSyncedEpisode = (&e) => { return e.oldId != null && ('' + e.oldId).startsWith('tmdb-episode:') }
600. static watchCountOf = (&id) => { return watchIds[id] != null ? watchIds[id].length : 0 }
601. // the most watches, else the synced one, else the first listed
602. static pickEpisodeKeeper = (&group) => {
603. let best = group[0]
604. for (e of group) {
605. let a = watchCountOf(e.id)
606. let b = watchCountOf(best.id)
607. if (a > b || (a == b && isSyncedEpisode(e) && !isSyncedEpisode(best))) { best = e }
608. }
609. return best
610. }
611. // episode `e` into `keeperId` (same season, same number)
612. static mergeEpisodeInto = (&e, &keeperId) => {
613. moveWatches(e.id, keeperId)
614. k = episodesTable.fetch(keeperId)
615. if (k == null) { return false }
616. x = k
617. if (isSyncedEpisode(e) && !isSyncedEpisode(k) && e.tmdbId != null) {
618. // a migrated keeper takes TMDB's current id and texts (its own kept, like the sync keeps a migrated row's)
619. if (k.tmdbId != null && k.migratedTmdbId == null) { x.migratedTmdbId = k.tmdbId }
620. x.tmdbId = e.tmdbId
621. if (!isEmpty(e.title) && e.title != k.title) {
622. if (!isEmpty(k.title) && k.migratedTitle == null) { x.migratedTitle = k.title }
623. x.title = e.title
624. }
625. if (!isEmpty(e.summary) && e.summary != k.summary) {
626. if (!isEmpty(k.summary) && k.migratedSummary == null) { x.migratedSummary = k.summary }
627. x.summary = e.summary
628. }
629. if (!isEmpty(e.release)) { x.release = e.release }
630. } else {
631. if (isEmpty(k.title) && !isEmpty(e.title)) { x.title = e.title }
632. if (isEmpty(k.summary) && !isEmpty(e.summary)) { x.summary = e.summary }
633. if (isEmpty(k.release) && !isEmpty(e.release)) { x.release = e.release }
634. }
635. if (isEmpty(k.tvmzId) && !isEmpty(e.tvmzId)) { x.tvmzId = e.tvmzId }
636. // the tombstone's TMDB id first (the table holds no id twice for the sync's lookup), then the keeper
637. y = episodesTable.fetch(e.id)
638. y.mergedInto = keeperId
639. y.mergedAt = now()
640. y.mergedTmdbId = e.tmdbId
641. y.tmdbId = null
642. if (episodesTable.update(y.id, y) == null) { console.log('merge: episode ' + y.id + ' not stored: ' + episodesTable.lastError()) }
643. if (episodesTable.update(x.id, x) == null) { console.log('merge: episode ' + x.id + ' not stored: ' + episodesTable.lastError()) }
644. forgetEpisodeKey(y.id)
645. forgetEpisodeKey(x.id)
646. return true
647. }
648. // one season: every number held twice or more → one episode (the others tombstones, out of the season's list)
649. static doubleEpisodesStep = (&seasonId, &showId) => {
650. se = seasonsTable.fetch(seasonId)
651. if (se == null || se.mergedInto != null) { return 0 }
652. byNum = {}
653. order = []
654. for (eid of listOr(se.episodes)) {
655. let e = episodesTable.fetch(eid)
656. if (e != null && e.mergedInto == null) {
657. let k = '' + e.episodeNumber
658. if (byNum[k] == null) {
659. byNum[k] = [e]
660. order.push(k)
661. } else { byNum[k].push(e) }
662. }
663. }
664. gone = {}
665. let n = 0
666. for (k of order) {
667. let group = byNum[k]
668. if (group.length > 1) {
669. let keeper = pickEpisodeKeeper(group)
670. for (e of group) {
671. if (e.id != keeper.id && mergeEpisodeInto(e, keeper.id)) {
672. gone[e.id] = true
673. n = n + 1
674. }
675. }
676. }
677. }
678. if (n == 0) { return 0 }
679. keep = []
680. parked = []
681. for (x of listOr(se.mergedEpisodes)) { parked.push(x) }
682. for (eid of listOr(se.episodes)) { if (gone[eid] == true) { parked.push(eid) } else { keep.push(eid) } }
683. z = se
684. z.episodes = keep
685. z.mergedEpisodes = parked
686. if (seasonsTable.update(z.id, z) == null) { console.log('merge: season ' + z.id + ' not stored: ' + seasonsTable.lastError()) }
687. totals.doubleSeasons = totals.doubleSeasons + 1
688. totals.doubleEpisodes = totals.doubleEpisodes + n
689. refreshCatalogShow(showId)
690. s = showsTable.fetch(showId)
691. console.log('merge: double episodes of "' + (s != null ? s.title : showId) + '" season ' + se.seasonNumber + ' ' + seasonId + ': ' + n + ' merged, ' + keep.length + ' left')
692. return n
693. }
695. // ONE item of the queue → { done, left }
696. static mergeStep = () => {
697. mergePrepare()
698. if (run.pos >= queue.length) { return { done = true left = 0 } }
699. item = queue[run.pos]
700. run.pos = run.pos + 1
701. if (item.kind == 'title') {
702. recs = []
703. for (id of item.ids) {
704. let r = showsTable.fetch(id)
705. if (r != null && r.mergedInto == null) { recs.push(r) }
706. }
707. if (recs.length > 1) {
708. keeper = pickKeeper(recs)
709. for (r of recs) { if (r.id != keeper) { mergeTitleInto(r.id, keeper) } }
710. totals.groups = totals.groups + 1
711. // the keeper's own stray seasons (naming it, not listed — Reacher's kind) the same way as any title's
712. strays[keeper] = listOr(seasonsByDup[keeper])
713. straySeasonsStep(keeper)
714. }
715. } else if (item.kind == 'episodes') {
716. doubleEpisodesStep(item.id, item.show)
717. } else {
718. straySeasonsStep(item.id)
719. }
720. return { done = run.pos >= queue.length left = queue.length - run.pos }
721. }