October 13, 2026 · About Cokpit

How Cokpit's hub keeps an agent session alive

A terminal session that lives in one window dies with it. Cokpit's hub moves every agent and shell session to a daemon on a home server, so a client can disconnect, come back from another machine and see a clean screen. This is how it works, what it took to build, and the two bugs worth remembering.

The problem

Working with AI coding agents means long-running terminal sessions. A Claude Code or Codex session can hold an hour of context, a half-finished refactor and a question waiting for an answer. In a stock terminal that session belongs to one window on one machine. Close the laptop, reboot the workstation, or walk to another desk and the session is gone, or at best it is still running somewhere you cannot see.

Kavish wanted three things: every session runs on a server that stays up, any of his machines can attach to it and detach without killing it, and attaching later shows the screen exactly as it is now, not a replay of everything that scrolled past.

Cokpit’s hub is the answer. It is a Rust daemon on a home Proxmox VM that owns every session, with Cokpit (his development environment) as its client.

The design

One daemon owns every session

Each session is a pseudo-terminal spawned by the daemon, not by the client. The client sends keystrokes over a WebSocket and receives output; if the client goes away, the process keeps running. Several clients can attach to one session at the same time and all see the same screen, which is what makes “attach from the laptop, finish on the workstation” work.

A cgroup per session, so kill means kill

Every session runs in its own cgroup v2 leaf. An agent CLI spawns shells, which spawn build tools, which spawn test runners; killing only the top process leaves orphans. Writing to the cgroup’s kill file reaps the whole tree at once. This needs a systemd user service with delegation enabled; a transient scope gets no delegated subtree, which cost a morning to learn.

Reattach is a snapshot of state, never a replay

This is the part that makes the hub feel right. The daemon runs a headless terminal emulator for every session and feeds it every byte the process emits, so the daemon always holds the authoritative screen. When a client attaches, the daemon renders that emulator state into one ANSI snapshot, sends it, and follows it with the live output stream from exactly the byte the snapshot ended on. The snapshot’s sequence number equals the first live chunk’s, taken under one lock, so the two can never gap or overlap.

The alternative, replaying the raw byte history, garbles the screen of any full-screen application and gets slower the longer a session lives. The snapshot approach is what the mature terminal multiplexers do, and it was chosen on purpose from the first day.

Getting the snapshot complete took a second pass. A full-screen agent runs on the alternate screen, so the snapshot has to carry both grids: the main screen and its scrollback behind the alternate one, then the switch, then the alternate grid. It also carries the scroll region, the saved cursor, the character sets, the full set of terminal modes an application may have switched on, the pending-wrap flag, the cursor’s pending text attributes and the window title. The test for it feeds one emulator, snapshots into a fresh second one, then feeds both the same bytes and asserts the screens stay identical.

The emulator library kept the fields the snapshot needs private. Rather than wrap seventy-odd handler methods and risk a silently missed forward, the library is vendored as a pinned path patch with five read-only accessors added. A moved field is a compile error, a missed forward is a silent no-op; the compile error is the better failure.

Two choices that look small

The PTY crate was chosen for how it closes. With the alternative, dropping the writer injects a newline and an end-of-file into the process, which would silently confirm a yes-or-no prompt in an agent that happened to be waiting on one.

The daemon deliberately does not speak the newer keyboard protocol some terminals advertise. Its emulator answers the agent’s capability queries on the client’s behalf, and the client’s terminal cannot honour those encodings, so advertising them would break keys instead of improving them.

Persistence and restart

Session records live in SQLite behind a single writer thread. When the daemon restarts, it marks every session exited, then auto-resumes the ones that were pinned, relaunching the agent with its own resume flag and the same conversation id. A pinned agent session survives a daemon upgrade with its context intact.

Reachable from anywhere, exposed nowhere

The daemon binds loopback only. The tailnet publishes one port to Kavish’s own devices and stamps each request with the owner’s identity; the daemon checks that header and a bearer token with a constant-time compare. Hooks and the reverse tunnel use separate loopback ports that are never published. “Loopback is trusted” proves nothing here, because the tailnet proxy reaches the backend from loopback too, which is why both gates exist.

What came after the core

The core shipped in late August 2026, built over one week as fourteen test-driven tasks with a whole-branch review at the end. Then the protocol grew in steps, each verified on the live daemon by the protocol version in its hello message rather than by a binary version string:

A jailed file API over the whole home directory, with a content hash as the etag, atomic temp-file-and-rename writes, and a directory watcher that pushes change events so an editor can reload a clean buffer in place and warn on a dirty one. Linux allows tens of thousands of watches per process but only 128 watcher instances, so the daemon runs one process-wide watcher, reference-counted per directory.
A bounded exec verb (no shell, closed stdin, a deadline, a process-group kill on timeout), a listening-ports scan read straight from the process table, and a tunnel primitive in both directions with a per-stream credit window, so the client can forward a dev server from the hub or let a hub-side agent reach the client’s own control plane.
Session awareness: a hook listener that agents report into, a record of which session is doing what, and a push of that record to every connected client.

The thin desktop client written for the hub was retired in September 2026 once Cokpit itself could do everything it did; one client, one code path.

Two bugs worth remembering

A reply lost in the close. From a hub session, the command-line client answered a request and closed its socket in the same breath. The daemon’s stream teardown aborted both of its tasks, including the writer still holding the reply, so the far side read nothing. The fix is a rule: a close with a peer still on the other end drains, bounded by a two-second timeout; only a dropped WebSocket aborts everything at once. The first version of the test for this passed against the broken code, because a fresh loopback socket on the hub has a 2.5 MiB send buffer and an entire 256 KiB window vanished into it. The test had to stall the reader deliberately and fill until credit stopped before it could see the bug.

A verb that never answered. An early control plane stalled every later verb whenever one busy pane held a lock across a blocking write. The fix was mechanical, but the lesson stuck: a control verb must answer with what actually happened, even when the request is unparseable, and a timed-out reply must never be handed to the next request in line.

By the numbers

The daemon’s test suite grew from 77 tests at the core’s first review to 509 at the session-awareness release in October 2026, every one running on the hub itself because the Windows workstation cannot build the cgroup code. The hub-side daemon and its first client together carried 484 commits and 197 daemon plus 318 client tests at acceptance. The daemon has been in daily use since late August 2026.

Implementation was drafted by AI coding agents from specs and plans Kavish wrote, under his review and tests. The design decisions above, and the two bugs, are the parts worth knowing before building something like it.