# Standing up an engine

Replicate komrad's AWS engines in your own account, then run them day to day.

This guide replicates [komrad’s engine](https://komrad.dev/latest/guides/komrads-engine.md) in an AWS account of your own. It builds the same architecture from an empty account, with each step in the order the later ones depend on, and then covers the routine jobs of running it. The Terraform, the Packer template and the mise tasks are the ones komrad’s own engines run on, so a different architecture means changing them first. Run every command from a komrad checkout. The mise tasks use the AWS profile named komrad unless your environment already has credentials.

## Prerequisites

A replica needs its own account with each service komrad’s engines depend on:

-   An AWS account with a named profile, and an S3 bucket for Terraform’s state.
-   An EC2 vCPU quota large enough for the control plane and an engine for each person.
-   The Session Manager plugin for the AWS CLI, which you need to get a shell on the control plane or an engine.
-   A Tailscale tailnet you administer, with Tailscale running on your laptop.
-   Komrad registered with your identity provider as a desktop app. [Komrad’s engine](https://komrad.dev/latest/guides/komrads-engine.md#provider-registration) covers the registration.
-   A GitHub OAuth app with device flow turned on, for people to authorize. Each organisation that restricts OAuth apps needs one of its owners to approve it.

## Building the images

```sh
mise run packer:build
```

The build makes the control plane image and the engine image, and prints each one’s id when it finishes. Terraform needs both ids in a later step. [Komrad’s engine](https://komrad.dev/latest/guides/komrads-engine.md#the-image) describes what each image contains.

## Preparing the tailnet

The control plane joins your tailnet as one tagged device, and the rest of the tailnet stays as it is. In Tailscale’s admin console:

1.  Under DNS, turn on MagicDNS and HTTPS certificates. The tailnet’s DNS name, such as `tail1234.ts.net`, is at the top of the page.
2.  In the access policy, add `tag:komrad-engine` to `tagOwners`, as [komrad’s engine](https://komrad.dev/latest/guides/komrads-engine.md#the-tailnet-policy) shows.
3.  Under Settings, then Keys, generate an auth key. Make it single-use and pre-approved, and tag it `tag:komrad-engine`. The control plane uses it only to join the first time, so a key that expires in a day is enough.

## Applying the infrastructure

Copy `terraform.tfvars.example` and `backend.hcl.example` in [`infra/engine`](https://github.com/harrisoncramer/komrad/tree/main/infra/engine), remove the `.example`, and fill them in. The variables include your account, the region, both image ids, the `sign_in_*` values from your provider and `github_oauth_client_id`, the client id of your GitHub OAuth app. Set `tailscale_hostname` to the control plane’s name on the tailnet, which defaults to komrad, and `tailscale_tailnet` to your tailnet’s DNS name, such as `tail1234.ts.net`. Set `tailscale_auth_key` to the auth key from the earlier step, which Terraform stores in Secrets Manager for the control plane. Add an engine for yourself, as [adding an engine](https://komrad.dev/latest/guides/standing-up-an-engine.md#adding-an-engine) shows. GitHub Actions needs `github_repository`, `github_owner_id`, `github_repository_id` and `state_bucket` to deploy. Then run:

```sh
mise run infra:engine:init
mise run infra:engine:apply
```

The apply prints each engine’s address, the instance ids and the CI role as `ci_role_arn`.

On first boot, the control plane joins the tailnet with the auth key, which spends the key, and starts Caddy. It also migrates the database and makes each person’s engine and grant. Each engine’s instance mounts its data volume, reads its engine login and starts its server.

Open a session on each instance and check that no unit failed. On a healthy instance the second command prints nothing:

```sh
aws ssm start-session --target i-0123456789abcdef0
systemctl list-units --state=failed --no-legend
```

## Configuring GitHub Actions

Komrad’s release and image workflows deploy to its engines. In your copy of the repository, they reach yours through these repository variables:

-   `AWS_CI_ROLE`, from the `ci_role_arn` output
-   `AWS_REGION`
-   `TF_STATE_BUCKET`
-   `TF_STATE_KEY`, matching the key in your backend file

If you changed the stack’s `name`, set `ENGINE_NAME` to match. The workflows skip their engine steps until `AWS_CI_ROLE` is set.

## Opening your engine

The control plane already granted your engine to your email. Check that your tailnet policy lets you reach the control plane on 443. Then, on your laptop, create a file in `~/.config/komrad/engines/` with your engine’s address from the apply:

```toml
[connection]
url = "https://komrad.tail1234.ts.net/you"
```

The file’s name is the engine’s name, so `dev.toml` makes an engine named `dev`. Sign in and open the panel:

```sh
komrad login --engine dev
komrad open --engine dev
```

The panel asks you to authorize GitHub the first time. In a shell on the engine, clone the repositories your agents work in by their https URLs. The forge proxy carries only https. Start each agent CLI once and sign it in. Then turn on [`trust_launch_dirs`](https://komrad.dev/latest/config.md#harness.claude.trust_launch_dirs) for each CLI in `~/.config/komrad/config.toml`, so new agents don’t stop to ask whether to trust their worktree. Restart the server afterwards, as [restarting](https://komrad.dev/latest/guides/standing-up-an-engine.md#restarting) shows.

## Adding an engine

Each person on the team needs an engine. Add the person to the engines in `terraform.tfvars`, with the email they sign in with and the instance’s size, and apply. The control plane makes the engine and grants it to that email. The person then opens their engine the same way you did.

A person also needs to reach the control plane over the tailnet. Check that your policy lets them reach it on 443, or share it with them from Tailscale’s admin console if they’re outside your tailnet.

To remove an engine, take the person out of `terraform.tfvars` and apply. Terraform deletes the instance and its volume, and the control plane removes the person’s grant.

## Running the engines

These jobs keep komrad’s engines running, and they work the same on a replica. Run them from a komrad checkout, with AWS credentials from the komrad profile or from your environment. The mise tasks look up the instances and their region in Terraform’s state.

### Getting a shell

You get a shell through AWS Systems Manager, which needs the Session Manager plugin for the AWS CLI. Look up an instance’s id in Terraform, then open a session:

```sh
mise run infra:engine:run output instance_ids
aws ssm start-session --target i-0123456789abcdef0
```

The session runs as a user with sudo. On an engine, switch to the komrad account to work as komrad and its agents do. That account owns the config, the worktrees and the CLIs’ logins:

```sh
sudo -u komrad -i
```

`mise run remote:connect harry` opens a session on the engine at `/harry` and switches to the komrad account in one step.

### Checking engine health

When an engine’s heartbeat stops, the dot beside its name in the panel turns red. Open a shell on the engine and list the failed units. On a healthy engine the command prints nothing:

```sh
systemctl list-units --state=failed --no-legend
```

Then look at the server in the same shell. The journal shows what systemd saw, like a crash at startup. Komrad’s log shows what the server wrote while it ran.

```sh
systemctl status komrad
journalctl -u komrad -n 200 --no-pager
tail -n 200 /var/lib/komrad/cache/komrad.log
```

If every engine is unreachable, open a shell on the control plane and look at Tailscale and Caddy. `tailscale status` shows whether the control plane is on the tailnet, and Caddy’s journal shows a certificate request that failed.

```sh
sudo tailscale status
systemctl status tailscaled caddy
journalctl -u caddy -n 200 --no-pager
```

To see the agents on an engine, list the windows on komrad’s tmux server as the komrad account:

```sh
tmux -L komrad list-windows -a
```

### Sign-in failures

When an engine rejects someone’s token, their panel asks them to sign in again:

```sh
komrad login --engine dev
```

If the engine still turns them away, check that the email on their engine in `terraform.tfvars` is the one they sign in with, and that their provider has verified it.

If their panel says the engine is unreachable, check that Tailscale runs on their laptop with DNS on and that the tailnet policy lets them in. If only their engine is unreachable, it may be stopped.

### Rejoining the tailnet

A replaced control plane keeps its tailnet identity through Parameter Store. If someone removes it from Tailscale’s admin console, it drops off the tailnet and needs a new key. Set `tailscale_auth_key` to a new single-use, pre-approved key tagged `tag:komrad-engine` and run `mise run infra:engine:apply`. Then join again in a shell on the control plane:

```sh
sudo systemctl restart komrad-tailscale
```

### Restarting

`mise run remote:restart harry` restarts the server of the engine at `/harry` and leaves its agents running. If the engine is stopped, it starts the engine instead.

`mise run remote:restart harry --hard` also kills the tmux server and creates a new manager session, which stops every agent on the engine. Save it for when the tmux session itself is broken.

People stay signed in through a restart, and their panels reconnect once the server is back.

### Deploying or rolling back

Releases deploy on their own. To deploy the build in your checkout, run `mise run remote:deploy`. One deploy updates the control plane and every running engine. A stopped engine keeps its last build until the first deploy after it starts.

To roll back, deploy an older ref. Either check it out and run `mise run remote:deploy`, or, without AWS credentials, dispatch `engine.yml` from the Actions tab on the tag or branch you want.

### Stopping and starting

Each person starts and stops their own engine from the panel, and every engine also stops each night. Stopping an engine stops every agent on it. The volume, the database and the address stay, and the person resumes the agents they need once it is back.

### Moving to a new image

Dispatch `image.yml` from the Actions tab to build both images. When it’s fine for the agents to stop, run:

```sh
mise run remote:reimage
```

The task lists the agents that will stop and asks you to confirm. It replaces the control plane and every engine’s instance, and keeps each engine’s volume. With no argument it uses the images `image.yml` recorded last. To use images you built yourself with `mise run packer:build`, pass their ids.

### Tearing it down

```sh
mise run infra:engine:destroy
```

Terraform can’t destroy the database while deletion protection is on, so set `database_deletion_protection = false` in `terraform.tfvars` and apply before you destroy. RDS takes a final snapshot of the database, and each engine’s data volume gets one too. Snapshot names have to be unique, so a second teardown needs a new snapshot name.

Then remove the control plane from Tailscale’s admin console and delete its tailnet identity. Terraform doesn’t own the parameter, and its path is `/<name>/tailscale/state` for the stack’s `name`:

```sh
aws ssm delete-parameter --name /komrad-development/tailscale/state
```
