komrad

Standing up an engine

Replicate komrad's AWS engines in your own account, then run them day to day.

Edit on GitHub

This guide replicates komrad’s engine in an AWS account of your own. It builds the same architecture from an empty account, with each step in the order the later ones depend on, and then covers the routine jobs of running it. The Terraform, the Packer template and the mise tasks are the ones komrad’s own engines run on, so a different architecture means changing them first. Run every command from a komrad checkout. The mise tasks use the AWS profile named komrad unless your environment already has credentials.

Prerequisites#

A replica needs its own account with each service komrad’s engines depend on:

  • An AWS account with a named profile, and an S3 bucket for Terraform’s state.
  • An EC2 vCPU quota large enough for the control plane and an engine for each person.
  • The Session Manager plugin for the AWS CLI, which you need to get a shell on the control plane or an engine.
  • A Tailscale tailnet you administer, with Tailscale running on your laptop.
  • Komrad registered with your identity provider as a desktop app. Komrad’s engine covers the registration.
  • A GitHub OAuth app with device flow turned on, for people to authorize. Each organisation that restricts OAuth apps needs one of its owners to approve it.

Building the images#

mise run packer:build

The build makes the control plane image and the engine image, and prints each one’s id when it finishes. Terraform needs both ids in a later step. Komrad’s engine describes what each image contains.

Preparing the tailnet#

The control plane joins your tailnet as one tagged device, and the rest of the tailnet stays as it is. In Tailscale’s admin console:

  1. Under DNS, turn on MagicDNS and HTTPS certificates. The tailnet’s DNS name, such as tail1234.ts.net, is at the top of the page.
  2. In the access policy, add tag:komrad-engine to tagOwners, as komrad’s engine shows.
  3. Under Settings, then Keys, generate an auth key. Make it single-use and pre-approved, and tag it tag:komrad-engine. The control plane uses it only to join the first time, so a key that expires in a day is enough.

Applying the infrastructure#

Copy terraform.tfvars.example and backend.hcl.example in infra/engine, remove the .example, and fill them in. The variables include your account, the region, both image ids, the sign_in_* values from your provider and github_oauth_client_id, the client id of your GitHub OAuth app. Set tailscale_hostname to the control plane’s name on the tailnet, which defaults to komrad, and tailscale_tailnet to your tailnet’s DNS name, such as tail1234.ts.net. Set tailscale_auth_key to the auth key from the earlier step, which Terraform stores in Secrets Manager for the control plane. Add an engine for yourself, as adding an engine shows. GitHub Actions needs github_repository, github_owner_id, github_repository_id and state_bucket to deploy. Then run:

mise run infra:engine:init
mise run infra:engine:apply

The apply prints each engine’s address, the instance ids and the CI role as ci_role_arn.

On first boot, the control plane joins the tailnet with the auth key, which spends the key, and starts Caddy. It also migrates the database and makes each person’s engine and grant. Each engine’s instance mounts its data volume, reads its engine login and starts its server.

Open a session on each instance and check that no unit failed. On a healthy instance the second command prints nothing:

aws ssm start-session --target i-0123456789abcdef0
systemctl list-units --state=failed --no-legend

Configuring GitHub Actions#

Komrad’s release and image workflows deploy to its engines. In your copy of the repository, they reach yours through these repository variables:

  • AWS_CI_ROLE, from the ci_role_arn output
  • AWS_REGION
  • TF_STATE_BUCKET
  • TF_STATE_KEY, matching the key in your backend file

If you changed the stack’s name, set ENGINE_NAME to match. The workflows skip their engine steps until AWS_CI_ROLE is set.

Opening your engine#

The control plane already granted your engine to your email. Check that your tailnet policy lets you reach the control plane on 443. Then, on your laptop, create a file in ~/.config/komrad/engines/ with your engine’s address from the apply:

[connection]
url = "https://komrad.tail1234.ts.net/you"

The file’s name is the engine’s name, so dev.toml makes an engine named dev. Sign in and open the panel:

komrad login --engine dev
komrad open --engine dev

The panel asks you to authorize GitHub the first time. In a shell on the engine, clone the repositories your agents work in by their https URLs. The forge proxy carries only https. Start each agent CLI once and sign it in. Then turn on trust_launch_dirs for each CLI in ~/.config/komrad/config.toml, so new agents don’t stop to ask whether to trust their worktree. Restart the server afterwards, as restarting shows.

Adding an engine#

Each person on the team needs an engine. Add the person to the engines in terraform.tfvars, with the email they sign in with and the instance’s size, and apply. The control plane makes the engine and grants it to that email. The person then opens their engine the same way you did.

A person also needs to reach the control plane over the tailnet. Check that your policy lets them reach it on 443, or share it with them from Tailscale’s admin console if they’re outside your tailnet.

To remove an engine, take the person out of terraform.tfvars and apply. Terraform deletes the instance and its volume, and the control plane removes the person’s grant.

Running the engines#

These jobs keep komrad’s engines running, and they work the same on a replica. Run them from a komrad checkout, with AWS credentials from the komrad profile or from your environment. The mise tasks look up the instances and their region in Terraform’s state.

Getting a shell#

You get a shell through AWS Systems Manager, which needs the Session Manager plugin for the AWS CLI. Look up an instance’s id in Terraform, then open a session:

mise run infra:engine:run output instance_ids
aws ssm start-session --target i-0123456789abcdef0

The session runs as a user with sudo. On an engine, switch to the komrad account to work as komrad and its agents do. That account owns the config, the worktrees and the CLIs’ logins:

sudo -u komrad -i

mise run remote:connect harry opens a session on the engine at /harry and switches to the komrad account in one step.

Checking engine health#

When an engine’s heartbeat stops, the dot beside its name in the panel turns red. Open a shell on the engine and list the failed units. On a healthy engine the command prints nothing:

systemctl list-units --state=failed --no-legend

Then look at the server in the same shell. The journal shows what systemd saw, like a crash at startup. Komrad’s log shows what the server wrote while it ran.

systemctl status komrad
journalctl -u komrad -n 200 --no-pager
tail -n 200 /var/lib/komrad/cache/komrad.log

If every engine is unreachable, open a shell on the control plane and look at Tailscale and Caddy. tailscale status shows whether the control plane is on the tailnet, and Caddy’s journal shows a certificate request that failed.

sudo tailscale status
systemctl status tailscaled caddy
journalctl -u caddy -n 200 --no-pager

To see the agents on an engine, list the windows on komrad’s tmux server as the komrad account:

tmux -L komrad list-windows -a

Sign-in failures#

When an engine rejects someone’s token, their panel asks them to sign in again:

komrad login --engine dev

If the engine still turns them away, check that the email on their engine in terraform.tfvars is the one they sign in with, and that their provider has verified it.

If their panel says the engine is unreachable, check that Tailscale runs on their laptop with DNS on and that the tailnet policy lets them in. If only their engine is unreachable, it may be stopped.

Rejoining the tailnet#

A replaced control plane keeps its tailnet identity through Parameter Store. If someone removes it from Tailscale’s admin console, it drops off the tailnet and needs a new key. Set tailscale_auth_key to a new single-use, pre-approved key tagged tag:komrad-engine and run mise run infra:engine:apply. Then join again in a shell on the control plane:

sudo systemctl restart komrad-tailscale

Restarting#

mise run remote:restart harry restarts the server of the engine at /harry and leaves its agents running. If the engine is stopped, it starts the engine instead.

mise run remote:restart harry --hard also kills the tmux server and creates a new manager session, which stops every agent on the engine. Save it for when the tmux session itself is broken.

People stay signed in through a restart, and their panels reconnect once the server is back.

Deploying or rolling back#

Releases deploy on their own. To deploy the build in your checkout, run mise run remote:deploy. One deploy updates the control plane and every running engine. A stopped engine keeps its last build until the first deploy after it starts.

To roll back, deploy an older ref. Either check it out and run mise run remote:deploy, or, without AWS credentials, dispatch engine.yml from the Actions tab on the tag or branch you want.

Stopping and starting#

Each person starts and stops their own engine from the panel, and every engine also stops each night. Stopping an engine stops every agent on it. The volume, the database and the address stay, and the person resumes the agents they need once it is back.

Moving to a new image#

Dispatch image.yml from the Actions tab to build both images. When it’s fine for the agents to stop, run:

mise run remote:reimage

The task lists the agents that will stop and asks you to confirm. It replaces the control plane and every engine’s instance, and keeps each engine’s volume. With no argument it uses the images image.yml recorded last. To use images you built yourself with mise run packer:build, pass their ids.

Tearing it down#

mise run infra:engine:destroy

Terraform can’t destroy the database while deletion protection is on, so set database_deletion_protection = false in terraform.tfvars and apply before you destroy. RDS takes a final snapshot of the database, and each engine’s data volume gets one too. Snapshot names have to be unique, so a second teardown needs a new snapshot name.

Then remove the control plane from Tailscale’s admin console and delete its tailnet identity. Terraform doesn’t own the parameter, and its path is /<name>/tailscale/state for the stack’s name:

aws ssm delete-parameter --name /komrad-development/tailscale/state