WebSocket & actors

Gateway join, presence, deck.line / deck.block.

This document specifies how actors (browser, worker) connect to a session gateway, exchange deck and direction text, and how workers persist memory in SQLite + vectors.

1. Session gateway

1.1 Transport

• URL: ws://<host>:<port> (e.g. ws://127.0.0.1:35987). No path; first message from client must be join and must include sessionId.
• Framing: each message is a JSON object (UTF-8 text frame). For high-volume token streams, clients may batch; gateway may reject frames > 256 KiB.

1.2 Join handshake

First message from client must be join (must include sessionId and actorId):

{
  "type": "join",
  "sessionId": "default",
  "actorId": "actor-uuid-123",
  "channelIds": ["default"],
  "skillIds": ["add_track", "adjust_instrument", "pattern_steps", "pattern_piano", "channel_mix", "master_mixer"]
}

Gateway responds:

{
  "type": "joined",
  "sessionId": "...",
  "you": { "actorId": "...", "clientId": "...", "channelIds": ["default"] },
  "actors": ["actor-1", "actor-xyz"],
  "replay": []
}

• actors: list of actorIds in the channel.
• actorId (required): for human joins, the gateway picks a paired agent lane when exactly one agent is online (v1 prefers ai-a); otherwise null. Browsers show “No agent” until an agent connects.
• When anyone joins or leaves, the gateway broadcasts presence: { "type": "presence", "sessionId": "...", "actors": ["agent-1", "human-xyz"] }.

Human deck stream (Play)

While the browser actor is playing, the browser sends:

1. control { "type": "control", "op": "playing_start", "actorId": "...", "authorId": "...", "perfStep": <host 16th> } — agents mark the session live and may run inference after buffered deck arrives.
2. deck.line per emitted deck line (throttled). The gateway stamps actorId from the connection and fans out.
3. control { "op": "playing_stop", ... } on stop — agents clear live mode.

Agents append deck.line text from the playing actor to a rolling buffer and respond with deck.stream_chunk then deck.block with their own actorId.

1.3 Message families

Error codes actually emitted by services/gateway/main.tish: BAD_JSON, JOIN_BAD, JOIN_FIRST. Codes such as SKILL_DENIED, LEASE_CONFLICT, PARSE_FAIL, RATE_LIMIT, and AUTH are aspirational — skill / ownership violations are currently handled by the receiver silently skipping disallowed lines (see DJ_SKILLS.md).

1.4 Ordering

• Gateway maintains seq per actorId (monotonic). All deck.line / deck.block / direct get a server timestamp + actor seq.
• Cross-lane order is not total; merge rules live in the browser (CO_DJ_SPACE.md).

1.5 Rooms

• One room per sessionId. All joined clients receive fanout for messages they are allowed to see (v1: fanout all).

2. Worker contract (actor + skills)

2.1 Process model

• One worker per actor per session (e.g. actor-1), or a pool with one job per (sessionId, actorId).
• Worker connects with actorId (e.g. actor-1), channelIds: ["default"], skillIds: [...].

2.2 Input loop

The persistence / RAG steps below (local DB, embeddings, chunks) are planned — see §3. Today the worker keeps an in-memory rolling context buffer only.

1. On each inbound deck.line / deck.block (other actors or merged snapshot): append to the context buffer (planned: also persist to local DB and embed chunks).
2. On each direct where targetActorId matches this actor's actorId: enqueue for response.
3. On timer or debounce: build context = system prompt + last N buffered messages (planned: + RAG top-k from chunks).

2.3 Output loop

The worker (services/agent-worker/main.tish) calls its LLM: handlePlayingStream and handleDirect snapshot the buffered peer stream into one prompt (llmLinesFrom), call callLLM, and stream the real model output — first as deck.stream_chunk (the finished text char-by-char), then as a committed deck.block. It falls back to a demo patch only when GRADIENT_MODEL_ACCESS_KEY (or MODEL_ACCESS_KEY) is unset or the call fails. The worker also ingests inbound deck.block from other actors into its context buffer so responses are aware of them. The old 3-second periodic demo deck.block flood has been removed.

Note: the worker streams the finished text char-by-char — provider-side SSE token streaming (live provider tokens) is planned, not implemented.

1. Snapshot the buffered peer stream into one prompt; call callLLM and stream the resulting text.
2. Emit deck.stream_chunk (per character of the finished output) for the UI typing indicator, then a deck.block.
3. Validate deck line against actor's skills (DJ_SKILLS.md); if invalid, log and skip or send error to gateway.
4. For deck.block: set effectivePerfStep to host perfStep + lookahead (default 64 sixteenths ≈ four 4/4 bars). Set submitDeadlinePerfStep to host step + slack (e.g. 48) so the host drops the block if it arrives too late. Omit both and use asap: true for emergency edits. Alternatively put @ perf_step N as the first line of deck (parsed by host; same as effectivePerfStep).

2.4 Reconnect

• Resend join with same actorId, channelIds, skillIds; gateway sends replay.
• Planned / not implemented in services/agent-worker: loading state from a SQLite messages table on reconnect. The worker is in-memory only — its context buffer does not survive a restart.

3. SQLite + vector schema (Planned)

Planned / not implemented in services/agent-worker. The worker is currently in-memory only: it holds a rolling context buffer in process and does not open SQLite, embed chunks, or run vector/RAG retrieval. The schema below is a forward-looking spec for agent memory, kept here as the target design.

Path: e.g. ~/.deckard-sessions/<sessionId>.sqlite (agent-local) or shared if gateway persists.

3.1 Tables (SQL)

CREATE TABLE messages (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  session_id TEXT NOT NULL,
  actor_id TEXT NOT NULL,
  role TEXT NOT NULL,  -- tpl | direct | system | tpl_out
  body TEXT NOT NULL,
  hub_seq INTEGER,
  ts INTEGER NOT NULL
);

CREATE TABLE tpl_snapshots (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  session_id TEXT NOT NULL,
  actor_id TEXT NOT NULL,
  tpl_text TEXT NOT NULL,
  ts INTEGER NOT NULL
);

CREATE TABLE chunks (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  session_id TEXT NOT NULL,
  source TEXT NOT NULL,  -- tpl | direct | doc
  text TEXT NOT NULL,
  ts INTEGER NOT NULL
);

-- Embedding storage: use sqlite-vec / sqlite-vss extension:
-- CREATE VIRTUAL TABLE chunk_embeddings USING vec0(
--   chunk_id INTEGER,
--   embedding FLOAT[1536]  -- dimension matches model
-- );

CREATE TABLE agent_memory (
  session_id TEXT PRIMARY KEY,
  summary TEXT,
  updated_ts INTEGER
);

3.2 Vector index

• sqlite-vec or sqlite-vss: store chunk_id + embedding; on ingest, chunk deck lines (~512 chars) and direct messages; query with same embedding model as ingest.
• Fallback: no extension — keyword search on chunks.text only.

3.3 Embedding model

• Fix one model per deployment (e.g. text-embedding-3-small, 1536-d). Document in agent .env.

4. Security notes

• Token on WS URL for v1 stub auth.
• direct channel: treat as user input; strip control chars; max length 8 KiB.
• Rate-limit deck.line per actorId (e.g. 30/sec).

5. Reference implementation layout

• services/gateway/ — Tish: npm run gateway (or tish run --features ws,process services/gateway/main.tish). Listens on ws://127.0.0.1:35987 (or CODJ_HUB_PORT).
• services/agent-worker/ — Tish agent: npm run agent (or tish run --features ws,http,fs,process services/agent-worker/main.tish). It calls its LLM (callLLM over the http feature) for real responses when GRADIENT_MODEL_ACCESS_KEY (or MODEL_ACCESS_KEY) is set, and falls back to a demo patch otherwise. Memory persistence and RAG (§2.4, §3) remain planned — the worker is in-memory only.

See repository package.json / README for run commands.

6. Troubleshooting

Agent gets "HTTP error: 200 OK" when connecting

The WebSocket gateway responds with 101 Switching Protocols, not 200. A 200 OK response means another process is handling the port (e.g. an old Node server or another HTTP server).

1. Start the gateway first: npm run gateway — you should see WebSocket server listening on ws://0.0.0.0:35987.
2. Check what is using the port: with the gateway running, run:

``bash lsof -i :35987 ` You should see a single process (the tish gateway). If you see another process (e.g. node`), stop it so only the gateway is listening.

1. Then start the agent: npm run agent.