Production runbook

Before enabling provisioning

1. Create a dedicated service account and persistent state directory, such as /var/lib/pve-agents.
2. Put secrets in a root-readable environment file, not the checkout. Set DATABASE_PATH to the persistent directory.
3. Bind to loopback or a trusted LAN address and terminate TLS before exposing the controller.
4. Set a 32+ character CONTROLLER_AUTH_SECRET and a CONTROLLER_URL that exactly matches the browser origin.
5. Create the GitHub OAuth app for operator sign-in and set the numeric CONTROLLER_OPERATOR_GITHUB_ID.
6. Create a GitHub App with Contents: read/write and Pull requests: read/write, install it for the repositories it may access, and configure its key.
7. Build a template, create the pool and API token, configure workspace networking, and add at least one agent in Settings → Agents.
8. Keep PROVISIONING_ENABLED=false until you have tested the infrastructure.

Create an API key for scripts with pnpm apikey:create <name>. It is shown once and stored only as a hash.

Deploy

Run the project's deployment script. Before starting a new release, run:

pnpm check && pnpm typecheck && pnpm test && pnpm build
pnpm db:migrate

db:migrate is required before serving traffic: migrations otherwise wait for the first database request. Verify GET /api/health after the service starts.

A minimal systemd unit is:

[Service]
User=pve-agents
WorkingDirectory=/opt/pve-agents
EnvironmentFile=/opt/pve-agents/.env
ExecStart=/usr/bin/pnpm start
Restart=on-failure

Set NODE_EXTRA_CA_CERTS directly in the unit if needed; Node reads it before the application loads the environment file.

Run workspaces

Use pnpm worker:tick for one worker pass, or pnpm worker:tick --watch 5 while validating a new installation. Once verified, set WORKER_ENABLED=true; the server then runs the same loop at WORKER_INTERVAL_SECONDS (default: 5).

Reaping is configured in Settings, not the environment. It will not remove blocked workspaces, workspaces with uncommitted or unpushed work, or ones that cannot be inspected.

Back up SQLite with the service stopped or using SQLite's online backup support. For file-level backups, preserve the database with its -wal and -shm files.

Reverse proxy

SSE must not be buffered or compressed. For Caddy:

reverse_proxy 10.0.3.50:3000 {
    flush_interval -1
}

@compressible not path /api/workspaces/*/agent
encode @compressible gzip

Use the controller's configured address, then test at the public hostname. A stream that connects but never delivers updates is usually being buffered by the proxy.