gitoriaLog in with ident

tracker

All repositories: gitoria

ReadmeCodePull requestsReleasesTicketsSettingsCommitdc40d859dc40d859tracker#31 (mission 026): duplicate titles merged — the 68 type+tmdbId pairs held by 157 records were the old tracker's (all migrated); merge.hl repair job (own clock, before the TMDB jobs) keeps one keeper per title (follows/watches > old short id > oldest), moves follows, watches, seasons, cast, credits, timelines, tombstones the rest (mergedInto, never deleted), slugs + short ids 301 to the keeper; stray seasons merged into their listed twin (Reacher S3 watches) or linked when watched; search import re-checks before its put; deploy.sh waits up to 90 s for 200; real copy 68 -> 0 dup ids, az5b2 follows/watches equal; gates 342/0, 32/0, 52/0mredc40d859/deploy.sh

9.1 KB

  1. #!/usr/bin/env bash
  2. # deploy.sh — tracker.worldapi.org: Loreana (this folder) → Byrodin. Run ON LOREANA.
  3. #
  4. # ./deploy.sh gates → backup → rsync → restart the container → public URL 200
  5. # ./deploy.sh --dry-run gates → rsync -n (shows what WOULD be sent), no backup, no restart, no URL check
  6. # ./deploy.sh --skip-tests skips the gates (LOUD warning) — only when you know why
  7. # ./deploy.sh --target DIR|HOST:DIR another destination (default below); a local DIR is
  8. # how the script is tested without touching Byrodin
  9. # ./deploy.sh --url URL the URL that must answer 200 after the restart (asked for up to 90 s)
  10. # (env: DEPLOY_TARGET, DEPLOY_URL, DEPLOY_SSH override the same defaults; DEPLOY_URL_WAIT the seconds the URL may take)
  11. # (backup: tars storage/.sessions/.env that exist on the target to Loreana's
  12. # /media/SLOW1TB2/deploy-backups/<app>/, keeps newest 5; --target DIR backs up DIR instead)
  13. #
  14. # THE FIRST DEPLOY is done by the architect on Byrodin (folder, .env with this app's ident
  15. # API key + secret — registered once in ident's /apps for origin https://tracker.worldapi.org
  16. # — nginx vhost, cert, DNS). This script only updates the CODE: storage/, .sessions/, .env,
  17. # .scratch/, server.*, tests/ fixtures and logs are never sent, so live data on Byrodin is
  18. # never touched. No --delete: a file removed here stays on Byrodin (harmless).
  19. set -euo pipefail
  21. # ---- the app ------------------------------------------------------------------------------
  22. APP=tracker.worldapi.org
  23. CONTAINER=tracker.worldapi.org
  24. [email protected]:/CONTAINERS/projects/tracker.worldapi.org
  25. DEFAULT_URL=https://tracker.worldapi.org/
  26. GATES=(
  27. "node tests/browser.mjs"
  28. "node tests/kinds.mjs"
  29. "node tests/franchises.mjs"
  30. )
  31. # NEVER SENT (rsync patterns; a leading / anchors at the app folder)
  32. EXCLUDES=(
  33. /.git/ /.gitignore
  34. /storage/ /.sessions/ /.env /.env.* /.scratch/ /server.* '*.log' '*.pid' node_modules/
  35. )
  36. # ---------------------------------------------------------------------------------------------
  38. DRY=0
  39. SKIP=0
  40. TARGET=${DEPLOY_TARGET:-$DEFAULT_TARGET}
  41. URL=${DEPLOY_URL:-$DEFAULT_URL}
  42. # Loreana's ssh config may not apply to Byrodin: -F /dev/null (ident STATUS)
  43. SSH=${DEPLOY_SSH:-ssh -F /dev/null -o BatchMode=yes -o ConnectTimeout=15}
  45. usage() { sed -n '2,15p' "$0" | sed 's/^# \{0,1\}//'; }
  46. while [ $# -gt 0 ]; do
  47. case "$1" in
  48. --dry-run) DRY=1 ;;
  49. --skip-tests) SKIP=1 ;;
  50. --target) TARGET=${2:?--target needs a value}; shift ;;
  51. --target=*) TARGET=${1#--target=} ;;
  52. --url) URL=${2:?--url needs a value}; shift ;;
  53. --url=*) URL=${1#--url=} ;;
  54. -h|--help) usage; exit 0 ;;
  55. *) echo "deploy: unknown argument: $1" >&2; usage >&2; exit 2 ;;
  56. esac
  57. shift
  58. done
  60. HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
  61. cd "$HERE"
  63. step() { printf '\n==> %s\n' "$*"; }
  64. run() { printf ' $ %s\n' "$*" >&2; "$@"; }
  65. die() { printf '\ndeploy: REFUSED — %s\n' "$*" >&2; exit 1; }
  67. # a target `host:dir` is remote, a plain path is local
  68. if [[ "$TARGET" == *:* ]]; then
  69. REMOTE_HOST=${TARGET%%:*}
  70. REMOTE_DIR=${TARGET#*:}
  71. else
  72. REMOTE_HOST=
  73. REMOTE_DIR=$TARGET
  74. fi
  76. step "[0/5] $APP → $TARGET (dry run: $DRY, skip tests: $SKIP)"
  77. echo " from $HERE"
  78. echo " url $URL"
  79. [ -x bin/hybriel ] || die "bin/hybriel is missing here"
  80. [ -f docker-compose.yml ] || die "docker-compose.yml is missing here"
  81. [ -f project.hl ] || die "project.hl is missing here"
  82. command -v rsync >/dev/null || die "rsync is not installed"
  84. # ---- 1. the gates ---------------------------------------------------------------------------
  85. if [ "$SKIP" = 1 ]; then
  86. step "[1/5] GATES SKIPPED"
  87. printf ' !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n'
  88. printf ' !! --skip-tests: NOTHING WAS TESTED. You deploy untested code. !!\n'
  89. printf ' !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n'
  90. else
  91. step "[1/5] gates (${#GATES[@]})"
  92. # headless Chromes that exist BEFORE the gates are not ours (other sessions' test runs)
  93. chromes() { ps -eo pid=,ppid=,args= | awk '/[h]l-browser-tier/ && /remote-debugging-port/ && !/--type=/ {print $1, $2}'; }
  94. before=" $(chromes | awk '{print $1}' | tr '\n' ' ') "
  95. for g in "${GATES[@]}"; do
  96. printf ' $ %s\n' "$g"
  97. out=$(mktemp)
  98. if ! $g > "$out" 2>&1; then
  99. grep -E '^FAIL|passed,' "$out" | sed 's/^/ /' || true
  100. echo " (full output: $out)"
  101. die "gate failed: $g"
  102. fi
  103. grep -E 'passed,' "$out" | tail -1 | sed 's/^/ /'
  104. rm -f "$out"
  105. done
  106. # a Chrome our gates LEFT: new since the gates began AND no longer owned by a running node test
  107. leaked=""
  108. while read -r pid ppid; do
  109. [ -z "$pid" ] && continue
  110. case "$before" in *" $pid "*) continue ;; esac
  111. ps -o args= -p "$ppid" 2>/dev/null | grep -q '^node\|/node ' && continue
  112. leaked="$leaked $pid"
  113. done < <(chromes)
  114. if [ -n "$leaked" ]; then
  115. die "a gate left a headless Chrome (pids:$leaked; ps -eo pid,args | grep hl-browser-tier)"
  116. fi
  117. fi
  119. # ---- 2. pre-deploy backup of the LIVE data (before anything is sent) ------------------------
  120. BACKUP_ROOT=${DEPLOY_BACKUP_ROOT:-/media/SLOW1TB2/deploy-backups}
  121. BACKUP_DIR="$BACKUP_ROOT/$APP"
  122. BACKUP_FILE="$BACKUP_DIR/$APP-$(date +%Y%m%d-%H%M).tgz"
  123. if [ "$DRY" = 1 ]; then
  124. step "[2/5] DRY RUN: backup skipped"
  125. else
  126. step "[2/5] backup live data ($APP) → $BACKUP_FILE"
  127. mkdir -p "$BACKUP_DIR"
  128. if [ -n "$REMOTE_HOST" ]; then
  129. present=$($SSH "$REMOTE_HOST" "cd '$REMOTE_DIR' 2>/dev/null && for p in storage .sessions .env; do [ -e \"\$p\" ] && echo \"\$p\"; done; true")
  130. else
  131. present=$(cd "$REMOTE_DIR" 2>/dev/null && for p in storage .sessions .env; do [ -e "$p" ] && echo "$p"; done; true)
  132. fi
  133. present=$(printf '%s' "$present" | tr '\n' ' ' | sed 's/ *$//')
  134. if [ -z "$present" ]; then
  135. echo " nothing to back up yet (no storage/.sessions/.env on the target)"
  136. else
  137. echo " taring: $present"
  138. if [ -n "$REMOTE_HOST" ]; then
  139. run $SSH "$REMOTE_HOST" "tar czf - -C '$REMOTE_DIR' $present" > "$BACKUP_FILE" \
  140. || { rm -f "$BACKUP_FILE"; die "backup failed (tar/ssh error) — refusing to deploy"; }
  141. else
  142. run tar czf "$BACKUP_FILE" -C "$REMOTE_DIR" $present \
  143. || { rm -f "$BACKUP_FILE"; die "backup failed (tar error) — refusing to deploy"; }
  144. fi
  145. [ -s "$BACKUP_FILE" ] || { rm -f "$BACKUP_FILE"; die "backup is empty — refusing to deploy"; }
  146. echo " $(du -h "$BACKUP_FILE" | cut -f1) $BACKUP_FILE"
  147. ls -1t "$BACKUP_DIR/$APP"-*.tgz 2>/dev/null | tail -n +6 | xargs -r rm -f --
  148. echo " keeping $(ls -1 "$BACKUP_DIR/$APP"-*.tgz 2>/dev/null | wc -l) archive(s) of $APP"
  149. fi
  150. fi
  152. # ---- 3. rsync the code ----------------------------------------------------------------------
  153. RSYNC=(rsync -az --no-owner --no-group --itemize-changes)
  154. for e in "${EXCLUDES[@]}"; do RSYNC+=("--exclude=$e"); done
  155. if [ -n "$REMOTE_HOST" ]; then
  156. RSYNC+=(-e "$SSH")
  157. else
  158. mkdir -p "$REMOTE_DIR"
  159. fi
  160. # the preview is ALWAYS made first (rsync -n): nothing that must stay on Byrodin may be in it
  161. step "[3/5] rsync preview (what would be sent)"
  162. preview=$(mktemp)
  163. run "${RSYNC[@]}" -n ./ "$TARGET/" > "$preview"
  164. sed 's/^/ /' "$preview"
  165. echo " ($(grep -c . "$preview" || true) lines)"
  166. if awk '{print $2}' "$preview" | grep -E '^(storage/|\.sessions/|\.env|\.scratch/|server\.)|\.log$|\.pid$' ; then
  167. die "the preview holds a path that must never be sent (see above)"
  168. fi
  169. rm -f "$preview"
  170. if [ "$DRY" = 1 ]; then
  171. step "[3/5] DRY RUN: nothing sent"
  172. else
  173. step "[3/5] rsync"
  174. run "${RSYNC[@]}" ./ "$TARGET/" | sed 's/^/ /'
  175. fi
  177. # ---- 4. restart the container ---------------------------------------------------------------
  178. RESTART="cd '$REMOTE_DIR' && docker compose up -d --build && docker compose restart && docker compose ps"
  179. if [ "$DRY" = 1 ]; then
  180. step "[4/5] DRY RUN: would restart $CONTAINER"
  181. if [ -n "$REMOTE_HOST" ]; then echo " would run: $SSH $REMOTE_HOST \"$RESTART\""; else echo " would run: bash -c \"$RESTART\""; fi
  182. else
  183. step "[4/5] restart $CONTAINER"
  184. if [ -n "$REMOTE_HOST" ]; then
  185. run $SSH "$REMOTE_HOST" "$RESTART"
  186. else
  187. run bash -c "$RESTART"
  188. fi
  189. fi
  191. # ---- 5. the URL answers 200 -----------------------------------------------------------------
  192. if [ "$DRY" = 1 ]; then
  193. step "[5/5] DRY RUN: would check $URL answers 200"
  194. step "dry run done — nothing was sent, nothing restarted"
  195. exit 0
  196. fi
  197. # tracker#31 (mission 026): the boot is slower now (the search index over ~100k people, the list caches, the slug map) — nginx
  198. # answers 502 until the app listens. Ask every 2 s for up to URL_WAIT seconds (default 90), print only changes of the answer.
  199. URL_WAIT=${DEPLOY_URL_WAIT:-90}
  200. step "[5/5] $URL must answer 200 (waiting up to ${URL_WAIT} s for the boot)"
  201. code=000
  202. last=
  203. start=$(date +%s)
  204. while :; do
  205. code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "$URL" || true)
  206. waited=$(( $(date +%s) - start ))
  207. if [ "$code" != "$last" ] || [ "$code" = 200 ]; then echo " after ${waited} s: $code"; last=$code; fi
  208. [ "$code" = 200 ] && break
  209. [ "$waited" -ge "$URL_WAIT" ] && break
  210. sleep 2
  211. done
  212. if [ "$code" != 200 ]; then
  213. if [ -n "$REMOTE_HOST" ]; then die "$URL answered $code, not 200, after ${URL_WAIT} s — look: $SSH $REMOTE_HOST docker logs --tail 50 $CONTAINER"; fi
  214. die "$URL answered $code, not 200, after ${URL_WAIT} s — look: docker logs --tail 50 $CONTAINER"
  215. fi
  216. step "deployed $APP → $TARGET, $URL answers 200"

Branches

Latest commits

  • dc40d859tracker#31 (mission 026): duplicate titles merged — the 68 type+tmdbId pairs held by 157 records were the old tracker's (all migrated); merge.hl repair job (own clock, before the TMDB jobs) keeps one keeper per title (follows/watches > old short id > oldest), moves follows, watches, seasons, cast, credits, timelines, tombstones the rest (mergedInto, never deleted), slugs + short ids 301 to the keeper; stray seasons merged into their listed twin (Reacher S3 watches) or linked when watched; search import re-checks before its put; deploy.sh waits up to 90 s for 200; real copy 68 -> 0 dup ids, az5b2 follows/watches equal; gates 342/0, 32/0, 52/0mre
  • 2667da05tracker#30 (mission 025): /my/ pages from slim cached title cards, episode rows and watch sets (after the jobs /my/series 1.8 s -> 0.06 s, /my/unwatched 4.1 -> 0.18 s); timeline page shows its name once; franchise widget under the poster/title; movies with TV leftovers (First Contact) go through the details repair; tools/count-tmdb-ids.hl; gates 327/0, 52/0, 32/0mre
  • 3909810dantcolony#40: mission references in README/STATUS/docs point to the moved missionsmre
  • e63b1d28antcolony#40: history (LOG.md), worker briefs (missions/) and reports moved here from antcolony, numbered per project; old numbers in antcolony docs/mission-map.mdmre
  • 1cda451dtracker: Hybriel master 190aa11d (#127 both shapes, GC correctness fc838894) — conductor adopts despite /my/series 2.4x after jobs (memory 8.0 → 1.7 GB boot); see reports/071mre
  • c1fa2f2etracker (mission 071): Hybriel master 190aa11d measured on the real copy vs the live binary 8590df63 — NOT adopted (after the first-start jobs /my/series 2.4x slower, /series 1.6x, RSS swings 7.4-12.2 GB; fresh it is flat at 1.7-2.2 GB and /my/unwatched faster), vendor stays 8efba065, candidate kept in .scratch/w071/vendor-190aa11d; tests/kinds.mjs: collection seed off (its TMDB request broke check 1 in 1 of 4 runs); tests/realdata-071.sh + realdata-071-bench.mjs + tools/realdata-071-table.py; README + STATUS (numbers, how to repeat); gates 325/0, 32/0, 50/0mre
  • 8081350atracker docs (mission 070): README (summary, Config HL_GC_BYTES — kept at Hybriel's default, the 256 MiB setting is taken out of docker-compose.yml again: the jobs grew to 12+ GB with it too, see STATUS), Test (three gates), Deploy (first start ~50 min: kinds then seed, restart once after collections done, memory numbers), Vendored Hybriel 8efba065 + #48 audit, Files; STATUS mission 070 entry (merges, migrated counts, lambda audit, gates, RSS old vs new, how to repeat, open points); docs/kinds.md + docs/franchises.md job order; tests/realdata-070-*.sh, tools/count-migrated.hl, tools/ref-params.py, tools/lambda-audit.pymre
  • 1ad19c8ctracker: re-vendor Hybriel master 8efba065 (#126 GC by bytes, #48 lambda parameters copy) (mission 070): bin/hybriel sha256 50361e95…, plugins core crypto data fetch fs http http1 mpackdb proc smtp time web; lambda audit: 13 lambdas change a passed record/list (11 through a local alias), no caller relies on it — unchanged; 319 read-only lambda parameters get & (no copy per call: /my/schedule 2.35 → 0.46 s, /my/unwatched 13.7 → 5.7 s on the real copy); /my/unwatched one merge sort instead of n² inserts; docker-compose HL_GC_BYTES=268435456; deploy.sh runs kinds.mjs + franchises.mjs too (default ports 8700–8710); tests/realdata-070.mjs; gates browser 325/0, kinds 32/0, franchises 50/0mre
  • 46b21389tracker (mission 070, conductor): the sync never destroys migrated data — the one-time summary step MOVES a copied summary to migratedSummary (marker summaries-moved.txt) instead of clearing it; a migrated record's first title/genres/homepage/tagline TMDB replaces → migratedTitle/migratedGenres/migratedHomepage/migratedTagline, a migrated season's/episode's title/summary → migratedTitle/migratedSummary (set once); tests/peek-shows.hl prints them; gate 325/0mre
  • 749019b0Merge t19 (tracker#19 franchises + timelines) into main (mission 070): conflicts README/STATUS/show.hl/project.hl/browser.mjs/faketmdb.mjs, both sides kept; franchise/timeline pages get #20's typed heading (Franchise | …, Timeline | …), their title links via titlePath (/movies|/series|/shows); the collection seed waits for repair, kinds and credits too; franchises.mjs URLs + 2 new checks; gates browser 323/0, kinds 32/0, franchises 50/0mre
  • daf49feaMerge t20 (tracker#20 typed headings + #21 series/shows split) into main (mission 070): conflicts README/STATUS/show.hl/project.hl/search.hl/components/search.hl/browser.mjs/faketmdb.mjs, both sides kept; clock order backfill → repair → kinds → credits; withDetailsFields stores tmdbType + kind; gates: browser.mjs URLs → /series|/movies, typed h1 selectors; kinds.mjs repair/credits off, fixture name = seed name; browser 323/0, kinds 32/0mre
  • 20e09d89Merge t18 (tracker#18 delta sync) into main (mission 070): conflicts README/STATUS/show.hl/project.hl/browser.mjs, both sides kept; pageShowOf summary = summaryOfmre
  • f83571c3tracker#18 (mission 067): daily sync by change lists — TMDB /tv|movie/changes (since the stored day, paged) + TVmaze /updates/shows → only our changed titles (followed: full step, unfollowed: light step — changed seasons, no TVmaze), full walk on first run / gap > 14 days / failed list; show record refreshed (title, tmdbSummary, tagline, status, genres …; renamed titles re-indexed); summary = the creator's own text (page: summary > tmdbSummary > tvmazeSummary), one-time clear of copied summaries (9,647 on the live copy); gate 261, tests/realdata-018*.mjs, README + STATUSmre
  • 1ed5457etracker#20 + #21 (mission 068): typed headings "<Type> | <name>" in type colours; TV titles split into Series (/series) and Shows (/shows) by TMDB type + Reality/Talk/News genres — kind stored by sync/import/adult backfill + new kind backfill (resumes), /movies/<slug>, /shows/<slug> of a series/movie → 301, /my/series + /my/shows, home 5 tiles + 3 rows, search/filmography labels; gates kinds 32 + browser 266, tests/realdata-068.mjs, tools/count-kinds.hl, docs/kinds.md, README + STATUSmre
  • f2b00674Merge t26 (tracker#26 + #28) into main (mission 062): short ids for every new person (castPersonId, guest route), guest stars stored on the title and created as people only when opened (/person/tmdb/<id>?show=<id> → 302), lean watch/follow clicks (showRow a small object, cast/crew from the slug, watches cached per user, face rows only after a season toggle); gate 311, tests/realdata-062.mjs, README + STATUSmre
  • 0553b51ftracker#19 (mission 066): franchises and timelines — tables, /franchises, /franchises/<slug>, /timelines/<slug> (Timeline | Release sort, series by last episode), the Prequel | Timeline | Sequel widget with the franchise above, the creator's editor, TMDB collection seed in the app (resumes, paced); gate tests/franchises.mjs 48/0 + browser.mjs 266/0, tests/realdata-066.mjs, docs/franchises.md, README + STATUSmre
  • fa1f9dfatracker#29 (mission 063): unwatched check muted grey outline + check (accent only on hover), watched stays solid — no code regression, the accent outline read as ticked; gate checks real checks visibly (computed style + screenshot pixel) on /my/unwatched, show, movie, /my/movies; gate 266, tests/realdata-063.mjs, README + STATUSmre
  • 10bb3f93tracker#28 (mission 061): full cast (all seasons, main cast by episodes, guest stars) + crew (created by, directed by, written by, screenplay, story, music) — stored by the details completion, the daily sync, the search import (one details request) and a background credits job (resumes, RSS limit); show page collapsed after 20 with client-side Show all; showBySlug via a slug map; gate 249, tests/realdata-028.mjs, README + STATUSmre
  • d8b12d67tracker#27 (mission 060): short ids for movies, series and persons — old 702 kept (data/old-short-ids.json), new random [a-z0-9]{5} unique across both, claimed at creation, background backfill (resumes), shown under poster/photo, /<shortId> → 301; gate 259, tests/realdata-060*, README + STATUSmre
  • 25a50bc4tracker#26 (mission 059): titles from a filmography are completed — on open (skeleton, step-wise face showComplete, no reload) and by the in-app details repair (resumes, TMDB-paced, series in parts); cast from TMDB credits; gate 231, tests/realdata-026.mjs, README + STATUSmre