gitoriaLog in with ident

tracker

All repositories: gitoria

ReadmeCodePull requestsReleasesTicketsSettingsCommitdc40d859dc40d859tracker#31 (mission 026): duplicate titles merged — the 68 type+tmdbId pairs held by 157 records were the old tracker's (all migrated); merge.hl repair job (own clock, before the TMDB jobs) keeps one keeper per title (follows/watches > old short id > oldest), moves follows, watches, seasons, cast, credits, timelines, tombstones the rest (mergedInto, never deleted), slugs + short ids 301 to the keeper; stray seasons merged into their listed twin (Reacher S3 watches) or linked when watched; search import re-checks before its put; deploy.sh waits up to 90 s for 200; real copy 68 -> 0 dup ids, az5b2 follows/watches equal; gates 342/0, 32/0, 52/0mredc40d859/plugins/http1/README.md

13.4 KB

  1. # hl:http1 Plugin
  3. HTTP/1.1 server plugin for Hybriel. Epoll-based acceptor with I/O thread pool, optional TLS, keep-alive support.
  5. ## Two Usage Patterns
  7. ### 1. Event-based (class with `on request`)
  9. Extend `NativeHttpServer`, override `on request`, call `listen()`.
  11. ```hybriel
  12. import { NativeHttpServer, Response } from 'hl:http1';
  14. MyServer < NativeHttpServer {
  15. Number port = 8091;
  17. on request(req) {
  18. return new Response('<!DOCTYPE>
  19. <html>
  20. <head>
  21. <title>Hello World!</title>
  22. </head>
  23. <body>
  24. <h1>Hello World!</h1>
  25. <p>This could be your front page!</p>
  26. </body>
  27. </html>
  28. ', {
  29. status = 200;
  30. headers = {
  31. "Set-Cookie" = 'session=test';
  32. };
  33. });
  34. }
  35. }
  37. myServer = { port = 8091; } > new MyServer();
  38. ```
  40. The server runs in the event loop. Each incoming request is dispatched to `on request` as a fiber, so the process stays alive and can handle concurrent requests.
  42. ### 2. Iterator-based (`for (req of server)`)
  44. Call `listen(port)` directly, iterate requests in a loop.
  46. ```hybriel
  47. import { listen, Response } from 'hl:http1';
  49. server = listen(9123);
  51. for (req of server) {
  52. if (req.path == "/hello") {
  53. emit req.response(new Response('<!DOCTYPE>
  54. <html>
  55. <head>
  56. <title>Hello World!</title>
  57. </head>
  58. <body>
  59. <h1>Hello World!</h1>
  60. <p>This could be your front page!</p>
  61. </body>
  62. </html>
  63. ', {
  64. status = 200;
  65. headers = {
  66. "Set-Cookie" = 'session=test';
  67. };
  68. }));
  69. } else {
  70. emit req.response(new Response('Not found', { status = 404; }));
  71. }
  72. }
  73. ```
  75. This blocks on each iteration waiting for the next request. Simpler but sequential.
  77. ## Request Object
  79. Each request has these fields:
  81. | Field | Type | Description |
  82. |-------|------|-------------|
  83. | `method` | String | `"GET"`, `"POST"`, etc. |
  84. | `path` | String | `"/hello"`, `"/api/users"` |
  85. | `query` | Object | Parsed query params (keys/values) |
  86. | `headers` | Object | HTTP headers (keys/values) |
  87. | `body` | String or null | Request body — a `Content-Length` one, or a `Transfer-Encoding: chunked` one de-chunked |
  88. | `bytes` | Bytes | The same body as a Bytes, byte for byte (empty when there is none) |
  89. | `respond` | Handle | Response handle |
  91. ## Response Object
  93. Create a `Response` with body and optional options:
  95. ```hybriel
  96. // Basic
  97. new Response("Hello!")
  99. // With status and headers
  100. new Response('{"ok":true}', {
  101. status = 200;
  102. headers = {
  103. "Content-Type" = 'application/json';
  104. "Set-Cookie" = 'session=abc';
  105. };
  106. })
  108. // Error response
  109. new Response('Not found', { status = 404; })
  110. ```
  112. | Field | Type | Default | Description |
  113. |-------|------|---------|-------------|
  114. | `body` | String or Bytes | `''` | Response body; a Bytes is sent raw, as `application/octet-stream` unless `headers` name a type |
  115. | `status` | Number | `200` | HTTP status code |
  116. | `headers` | Object | `{}` | HTTP headers (key-value pairs) |
  118. ## NativeHttpServer Properties
  120. | Property | Type | Default | Description |
  121. |----------|------|---------|-------------|
  122. | `port` | Number | 8080 | Listen port |
  123. | `host` | String | "0.0.0.0" | Bind address |
  124. | `tls_cert` | String | null | Path to TLS certificate |
  125. | `tls_key` | String | null | Path to TLS private key |
  126. | `threads` | Number | 4 | I/O worker threads |
  128. ## Architecture
  130. ```
  131. Client → epoll acceptor thread → PARK (epoll) → I/O thread pool (TLS + parse)
  132. → request queue → event loop → on request fiber
  133. ```
  135. - Acceptor thread uses epoll to accept connections without blocking
  136. - I/O workers handle TLS handshake and HTTP parsing in parallel
  137. - Parsed requests go into an MPSC queue
  138. - The runtime event loop polls the queue (non-blocking) and spawns a fiber per request
  140. ### Parking: an idle connection costs no thread (mission 084)
  142. A worker's `read()` is blocking, so a connection handed to the pool occupies a whole
  143. worker until bytes arrive. Keep-alive connections used to be re-enqueued straight into the
  144. pool after each response, so an idle one still held a thread — with the default
  145. `threads = 4`, **four quiet sockets starved the server** and every later request hung
  146. (mission 075 saw this as "the second browser session's page never fires `load`").
  148. Connections are therefore **parked** in a dedicated epoll instance (`ParkedConns`) and only
  149. enter the worker queue once they are actually readable — or have hung up, which a worker
  150. reads as EOF and closes. Both the acceptor and the post-response keep-alive path park.
  151. An idle connection now costs one fd and zero threads.
  153. - A parked connection that stays silent for `PARK_IDLE_TIMEOUT_MS` (60s) is closed.
  154. - Workers set `SO_RCVTIMEO` (30s) on every connection as a backstop against a client that
  155. trickles headers; it is no longer the idle-timeout mechanism.
  156. - **An established TLS connection is never parked**: OpenSSL may hold already-decrypted
  157. plaintext that epoll on the raw fd cannot see. Those go straight to a worker, as before,
  158. so a TLS server is still starvable by idle keep-alive connections — the plain-HTTP path
  159. (and the pre-handshake TLS socket, whose ClientHello does arrive on the raw fd) is fixed.
  160. - Regression test: `tests/browser/tests/13-keepalive-starvation.mjs`.
  162. ### WebSocket liveness sweep (mission 091)
  164. The WS reader loop (one thread, epoll, level-triggered EPOLLIN) already wakes on a 500ms
  165. timeout, so liveness costs no timer and no extra thread: on every tick it walks the
  166. upgraded sockets, pings the ones that have gone quiet, and drops the ones whose pong is
  167. overdue.
  169. A drop enqueues the **same `close` event a real FIN would have produced**, so every
  170. consumer above this plugin — hl:web's push subscription registry included — prunes through
  171. the path it already had. No new concept travels upward, and nothing at the `.hl` level
  172. needs a timer facility.
  174. Why it is needed: a peer that vanishes without closing (dead wifi, a suspended laptop, a
  175. host that never sent the FIN) leaves a socket that stays open and **writable**. A server
  176. with nothing to push at it never discovers the corpse, so before this the subscription was
  177. immortal. A closed tab is not this case — the browser's stack sends the FIN, and even under
  178. CDP offline emulation it still answers protocol pings (measured, mission 091), which is why
  179. the regression test's dead peer is a raw socket that goes silent rather than a tab.
  181. | Env var | Default | Meaning |
  182. |---------|---------|---------|
  183. | `HL_WS_PING_MS` | `15000` | a socket quiet this long is pinged |
  184. | `HL_WS_PONG_TIMEOUT_MS` | `10000` | a ping unanswered this long ⇒ the peer is gone |
  186. Read once when the engine starts; timeouts are measured on `CLOCK_MONOTONIC`, so an NTP
  187. step cannot make one fire early or never. Any inbound frame — of any opcode — counts as
  188. proof of life and clears an outstanding ping.
  190. - Regression test: `tests/browser/tests/23-ws-liveness-sweep.mjs` (includes the
  191. falsification: the same silent peer against a sweep-disabled server stays registered).
  193. ### The handshake's cookies + a cookie-grade random (mission 093)
  195. A WebSocket upgrade is an ordinary HTTP request, so the browser attaches the site's cookies
  196. to it. That is the one moment an upgraded socket can be attributed to whoever loaded the
  197. page — after the 101 the request and its headers are freed and the connection carries no
  198. identity of its own. The `connect` event therefore gained a sixth field:
  200. ```
  201. { kind, id, data, binary, code, cookie }
  202. ```
  204. `cookie` is the handshake's `Cookie:` header verbatim, non-empty on `connect` only. Nothing
  205. is parsed here: hl:web's session layer (`plugins/web/web_session.hl`) owns the cookie's name
  206. and meaning, and this plugin owns only its delivery.
  208. ```
  209. randomToken(Number n) // n lowercase hex chars, kernel CSPRNG (getrandom(2)), max 128
  210. ```
  212. A session id is the only thing between a stranger and someone else's session, so it may not
  213. come from a seeded PRNG — `hl:math`'s `random()` is a clock-seeded xoshiro whose state a
  214. handful of outputs reveals, which would make every other id derivable from one's own. This
  215. lives here because a session cookie is an HTTP artifact and this is the plugin that parses
  216. and writes one; it is currently the only CSPRNG reachable from `.hl`.
  218. - Regression test: `tests/browser/tests/25-sessions.mjs` (two real browser instances = two
  219. cookie jars = two sessions).
  221. Ticket #74 added `host` (the handshake's `Host:` header) and ticket #105 `headers`: every
  222. header of the handshake as `name: value` lines, names lowercase, non-empty on `connect` only.
  223. `NativeWebSocketServer` hands them to the `WsClient` as `client.headers`, a hash by name —
  224. which is where hl:web reads the headers a navigation over that socket is constructed with.
  226. ## The outgoing WebSocket client (ticket #109)
  228. Everything above is the SERVER half — a browser (or anything else) dialling in.
  229. `WebSocketClient` is the other direction: Hybriel dialling OUT to someone else's
  230. WebSocket server. Subclass it and answer its three events. A subclass does NOT inherit
  231. the parent's run body (same rule as `NativeHttpServer`/`NativeWebSocketServer`: a parent's
  232. own root never runs for it) — end your own file's root with `connect()`, exactly as those
  233. two end theirs with `listen()`:
  235. ```hybriel
  236. import { WebSocketClient } from 'hl:http1'
  237. inherit WebSocketClient
  239. String url = "wss://example.com/socket"
  240. Hybrid headers = { "Authorization" = "Bearer …" }
  242. on open() {
  243. send('hello')
  244. }
  246. on message(text) {
  247. console.log('got: ' + text)
  248. }
  250. on close(code) {
  251. console.log('closed: ' + code)
  252. }
  254. connect()
  255. ```
  257. Or attach handlers to an instance directly, without subclassing at all — the same
  258. `on instance.event(...)` form used throughout this plugin's own tests:
  260. ```hybriel
  261. import { WebSocketClient } from 'hl:http1'
  263. client = new WebSocketClient(url = 'ws://127.0.0.1:8091/')
  265. on client.open() { client.send('hello') }
  266. on client.message(text) { console.log('got: ' + text) }
  267. on client.close(code) { console.log('closed: ' + code) }
  268. ```
  270. ### Properties
  272. | Property | Type | Default | Description |
  273. |----------|------|---------|-------------|
  274. | `url` | String | `null` | `ws://host[:port][/path]` or `wss://…` |
  275. | `headers` | Object | `{}` | Extra headers sent on the Upgrade request |
  276. | `caFile` | String | `null` | `wss://` only — an EXTRA trust anchor for a self-signed fixture, exactly hl:fetch's field of the same name (never a replacement for the system trust store) |
  277. | `connected` | Boolean | `false` | True between `open` and `close` |
  279. ### Methods
  281. | Method | Answers | Description |
  282. |--------|---------|-------------|
  283. | `connect()` | `true`/`false` | Dial `url` (idempotent — a no-op while already dialling or open). The file root calls this for you. |
  284. | `send(text)` | `1`/`0` | One text frame to the peer; `0` when not open |
  285. | `close([code])` | `1`/`0` | Start the close handshake; `close` still fires once the peer's echo (or a timeout) completes it |
  287. ### Events
  289. ```
  290. emit open() // the connection is open and ready for send()
  291. emit message(String data) // one complete text (or binary) message
  292. emit close(Number code) // the connection ended — RFC 6455 §7.4 code
  293. ```
  295. At most one `open` ever precedes a `close`: a connection that never opened still gets
  296. exactly one `close` (`1006` for a dial that never reached a peer — DNS failure, refused,
  297. TLS handshake failure; `1002` for a handshake the peer answered but got wrong).
  299. ### `wss://` uses hl:fetch's TLS
  301. The client half of `plugins/http/tls_common.zig` — the SAME module hl:fetch uses for
  302. `https://`. The system trust store, hostname verification always on, and `caFile` for an
  303. extra anchor. There is no verify-off switch, on this client either.
  305. ### Ping/pong is automatic; reconnect never is
  307. A quiet connection is pinged after `HL_WS_PING_MS` and dropped if the pong doesn't land
  308. within `HL_WS_PONG_TIMEOUT_MS` — the SAME two env vars the server's liveness sweep reads
  309. (mission 091, above). Any inbound ping is answered with a pong immediately. Neither ever
  310. reaches `.hl`: the protocol requires both, and they are not application information.
  312. **Reconnect is never automatic.** A dropped connection reaches you as exactly one `close`
  313. event and nothing here dials again on its own — call `connect()` again yourself (typically
  314. from your own `on close`) if you want a reconnect policy; `connect()` after a `close` opens
  315. a fresh connection, it is not a one-shot object.
  317. ### Architecture
  319. One dedicated thread per connection — the same shape as hl:fetch's one-thread-per-ticket
  320. model, not the server's shared epoll pool: a client dials a handful of long-lived peers,
  321. not thousands of short ones, and a private thread means a private event queue with no risk
  322. of misdelivering one connection's frames onto another instance's `on open`/`on
  323. message`/`on close`. The thread does DNS + TCP connect, the TLS handshake for `wss://`, the
  324. HTTP Upgrade handshake, then a read loop over the shared RFC 6455 framing
  325. (`plugins/http/ws_common.zig`) — this side's own frames go out through
  326. `writeFrameMasked`/`writeCloseMasked` (RFC 6455 §5.1: "a client MUST mask all frames"),
  327. freshly masked per frame from the kernel CSPRNG.
  329. - Fixtures: `tests/pass/plugins/039_http1_ws_client.hl` (open, send, a server-pushed
  330. frame, close from the client) and
  331. `tests/pass/plugins/040_http1_ws_client_close_reconnect.hl` (close from the SERVER's
  332. side, and an explicit reconnect proving it is never automatic).

Branches

Latest commits

  • dc40d859tracker#31 (mission 026): duplicate titles merged — the 68 type+tmdbId pairs held by 157 records were the old tracker's (all migrated); merge.hl repair job (own clock, before the TMDB jobs) keeps one keeper per title (follows/watches > old short id > oldest), moves follows, watches, seasons, cast, credits, timelines, tombstones the rest (mergedInto, never deleted), slugs + short ids 301 to the keeper; stray seasons merged into their listed twin (Reacher S3 watches) or linked when watched; search import re-checks before its put; deploy.sh waits up to 90 s for 200; real copy 68 -> 0 dup ids, az5b2 follows/watches equal; gates 342/0, 32/0, 52/0mre
  • 2667da05tracker#30 (mission 025): /my/ pages from slim cached title cards, episode rows and watch sets (after the jobs /my/series 1.8 s -> 0.06 s, /my/unwatched 4.1 -> 0.18 s); timeline page shows its name once; franchise widget under the poster/title; movies with TV leftovers (First Contact) go through the details repair; tools/count-tmdb-ids.hl; gates 327/0, 52/0, 32/0mre
  • 3909810dantcolony#40: mission references in README/STATUS/docs point to the moved missionsmre
  • e63b1d28antcolony#40: history (LOG.md), worker briefs (missions/) and reports moved here from antcolony, numbered per project; old numbers in antcolony docs/mission-map.mdmre
  • 1cda451dtracker: Hybriel master 190aa11d (#127 both shapes, GC correctness fc838894) — conductor adopts despite /my/series 2.4x after jobs (memory 8.0 → 1.7 GB boot); see reports/071mre
  • c1fa2f2etracker (mission 071): Hybriel master 190aa11d measured on the real copy vs the live binary 8590df63 — NOT adopted (after the first-start jobs /my/series 2.4x slower, /series 1.6x, RSS swings 7.4-12.2 GB; fresh it is flat at 1.7-2.2 GB and /my/unwatched faster), vendor stays 8efba065, candidate kept in .scratch/w071/vendor-190aa11d; tests/kinds.mjs: collection seed off (its TMDB request broke check 1 in 1 of 4 runs); tests/realdata-071.sh + realdata-071-bench.mjs + tools/realdata-071-table.py; README + STATUS (numbers, how to repeat); gates 325/0, 32/0, 50/0mre
  • 8081350atracker docs (mission 070): README (summary, Config HL_GC_BYTES — kept at Hybriel's default, the 256 MiB setting is taken out of docker-compose.yml again: the jobs grew to 12+ GB with it too, see STATUS), Test (three gates), Deploy (first start ~50 min: kinds then seed, restart once after collections done, memory numbers), Vendored Hybriel 8efba065 + #48 audit, Files; STATUS mission 070 entry (merges, migrated counts, lambda audit, gates, RSS old vs new, how to repeat, open points); docs/kinds.md + docs/franchises.md job order; tests/realdata-070-*.sh, tools/count-migrated.hl, tools/ref-params.py, tools/lambda-audit.pymre
  • 1ad19c8ctracker: re-vendor Hybriel master 8efba065 (#126 GC by bytes, #48 lambda parameters copy) (mission 070): bin/hybriel sha256 50361e95…, plugins core crypto data fetch fs http http1 mpackdb proc smtp time web; lambda audit: 13 lambdas change a passed record/list (11 through a local alias), no caller relies on it — unchanged; 319 read-only lambda parameters get & (no copy per call: /my/schedule 2.35 → 0.46 s, /my/unwatched 13.7 → 5.7 s on the real copy); /my/unwatched one merge sort instead of n² inserts; docker-compose HL_GC_BYTES=268435456; deploy.sh runs kinds.mjs + franchises.mjs too (default ports 8700–8710); tests/realdata-070.mjs; gates browser 325/0, kinds 32/0, franchises 50/0mre
  • 46b21389tracker (mission 070, conductor): the sync never destroys migrated data — the one-time summary step MOVES a copied summary to migratedSummary (marker summaries-moved.txt) instead of clearing it; a migrated record's first title/genres/homepage/tagline TMDB replaces → migratedTitle/migratedGenres/migratedHomepage/migratedTagline, a migrated season's/episode's title/summary → migratedTitle/migratedSummary (set once); tests/peek-shows.hl prints them; gate 325/0mre
  • 749019b0Merge t19 (tracker#19 franchises + timelines) into main (mission 070): conflicts README/STATUS/show.hl/project.hl/browser.mjs/faketmdb.mjs, both sides kept; franchise/timeline pages get #20's typed heading (Franchise | …, Timeline | …), their title links via titlePath (/movies|/series|/shows); the collection seed waits for repair, kinds and credits too; franchises.mjs URLs + 2 new checks; gates browser 323/0, kinds 32/0, franchises 50/0mre
  • daf49feaMerge t20 (tracker#20 typed headings + #21 series/shows split) into main (mission 070): conflicts README/STATUS/show.hl/project.hl/search.hl/components/search.hl/browser.mjs/faketmdb.mjs, both sides kept; clock order backfill → repair → kinds → credits; withDetailsFields stores tmdbType + kind; gates: browser.mjs URLs → /series|/movies, typed h1 selectors; kinds.mjs repair/credits off, fixture name = seed name; browser 323/0, kinds 32/0mre
  • 20e09d89Merge t18 (tracker#18 delta sync) into main (mission 070): conflicts README/STATUS/show.hl/project.hl/browser.mjs, both sides kept; pageShowOf summary = summaryOfmre
  • f83571c3tracker#18 (mission 067): daily sync by change lists — TMDB /tv|movie/changes (since the stored day, paged) + TVmaze /updates/shows → only our changed titles (followed: full step, unfollowed: light step — changed seasons, no TVmaze), full walk on first run / gap > 14 days / failed list; show record refreshed (title, tmdbSummary, tagline, status, genres …; renamed titles re-indexed); summary = the creator's own text (page: summary > tmdbSummary > tvmazeSummary), one-time clear of copied summaries (9,647 on the live copy); gate 261, tests/realdata-018*.mjs, README + STATUSmre
  • 1ed5457etracker#20 + #21 (mission 068): typed headings "<Type> | <name>" in type colours; TV titles split into Series (/series) and Shows (/shows) by TMDB type + Reality/Talk/News genres — kind stored by sync/import/adult backfill + new kind backfill (resumes), /movies/<slug>, /shows/<slug> of a series/movie → 301, /my/series + /my/shows, home 5 tiles + 3 rows, search/filmography labels; gates kinds 32 + browser 266, tests/realdata-068.mjs, tools/count-kinds.hl, docs/kinds.md, README + STATUSmre
  • f2b00674Merge t26 (tracker#26 + #28) into main (mission 062): short ids for every new person (castPersonId, guest route), guest stars stored on the title and created as people only when opened (/person/tmdb/<id>?show=<id> → 302), lean watch/follow clicks (showRow a small object, cast/crew from the slug, watches cached per user, face rows only after a season toggle); gate 311, tests/realdata-062.mjs, README + STATUSmre
  • 0553b51ftracker#19 (mission 066): franchises and timelines — tables, /franchises, /franchises/<slug>, /timelines/<slug> (Timeline | Release sort, series by last episode), the Prequel | Timeline | Sequel widget with the franchise above, the creator's editor, TMDB collection seed in the app (resumes, paced); gate tests/franchises.mjs 48/0 + browser.mjs 266/0, tests/realdata-066.mjs, docs/franchises.md, README + STATUSmre
  • fa1f9dfatracker#29 (mission 063): unwatched check muted grey outline + check (accent only on hover), watched stays solid — no code regression, the accent outline read as ticked; gate checks real checks visibly (computed style + screenshot pixel) on /my/unwatched, show, movie, /my/movies; gate 266, tests/realdata-063.mjs, README + STATUSmre
  • 10bb3f93tracker#28 (mission 061): full cast (all seasons, main cast by episodes, guest stars) + crew (created by, directed by, written by, screenplay, story, music) — stored by the details completion, the daily sync, the search import (one details request) and a background credits job (resumes, RSS limit); show page collapsed after 20 with client-side Show all; showBySlug via a slug map; gate 249, tests/realdata-028.mjs, README + STATUSmre
  • d8b12d67tracker#27 (mission 060): short ids for movies, series and persons — old 702 kept (data/old-short-ids.json), new random [a-z0-9]{5} unique across both, claimed at creation, background backfill (resumes), shown under poster/photo, /<shortId> → 301; gate 259, tests/realdata-060*, README + STATUSmre
  • 25a50bc4tracker#26 (mission 059): titles from a filmography are completed — on open (skeleton, step-wise face showComplete, no reload) and by the in-app details repair (resumes, TMDB-paced, series in parts); cast from TMDB credits; gate 231, tests/realdata-026.mjs, README + STATUSmre