# Operations

Runbooks for the people who run ADapptive. Deployment mechanics are in
`deploy/README.md`; this page covers everything after "it is deployed".

## Production map

| Service | Host | Health |
|---|---|---|
| Dashboard + public site | `app.adapptive.space` (apex `adapptive.space` redirects) | `GET /` 200; `GET /llms.txt` 200 |
| API | `api.adapptive.space` | `GET /health` → `{"status":"ok","database":"ok"}` |
| Remote MCP | `mcp.adapptive.space` | `GET /health` 200; unauthenticated `POST /mcp` → 401 with `WWW-Authenticate` |
| Authentik | `auth.adapptive.space` | `GET /-/health/live/` |
| Postgres 17, MinIO | private network | compose healthchecks |
| Umami | shared hs2 resource | — |

Coolify auto-deploys `main`. CI (`.github/workflows/ci.yml`) must be green
before merging to `main`.

## Deploy checklist

1. CI green on the PR.
2. Migrations are additive and reviewed; destructive changes need a two-step
   plan (deploy code that tolerates both shapes, then migrate).
3. New env vars added to Coolify **before** the deploy (see the variable table
   in `deploy/README.md`; new since the level-up: `SAFETY_CLASSIFIER`,
   `SAFETY_DAILY_BUDGET_USD`, `TRUSTED_ACCOUNT_MIN_AGE_HOURS`,
   `MCP_PUBLIC_URL`, `X402_*`, `HOUSE_POOL_AUTO_APPROVE`, `SENTRY_DSN`).
4. After deploy: probe the health table above; open the dashboard; run one
   `get_ready_feedback` against the remote MCP with a test key.

## Apex domain (one-time operator step)

In Coolify → the dashboard service → Domains: `https://adapptive.space,https://app.adapptive.space`.
Traefik provisions certificates for both; the Next.js host-based redirect
sends apex traffic to `app.adapptive.space` with a 308.

## Authentik (operator steps)

- Dashboard client and extension public client: `deploy/README.md`.
- **Tester enrollment**: create an enrollment flow bound to the tester
  application slug (`adapptive-testers`), with email verification and a
  captcha stage. Set `OIDC_AUDIENCE` to include its client id. Leave disabled
  for closed beta.
- Rotate the signing key yearly; the API reads JWKS at runtime.

## Backups

- **Postgres**: `deploy/backup.sh` runs `pg_dump --format=custom` and pushes
  to a restic repository off-host; retention `--keep-daily 7 --keep-weekly 5
  --keep-monthly 12`. Schedule it as a Coolify scheduled task (daily 03:00
  UTC) on the `postgres` service or a sidecar with the restic binary.
- **MinIO**: `mc mirror` the `adapptive-screenshots` bucket into the same
  restic repository weekly, or enable bucket versioning plus lifecycle rules.
- **Restore drill** (quarterly): restore the latest dump into
  `adapptive_restore_test`, run `pnpm --filter @adapptive/api test` against it
  with `SEED_RESET` disabled, record the time-to-restore.

## Error tracking

GlitchTip (Sentry-compatible) runs as its own Coolify resource
(`deploy/docker-compose.glitchtip.yml`). Set `SENTRY_DSN` on the API,
dashboard, and MCP services; the SDKs are initialised only when the DSN is
present. Alert routing: GlitchTip → email/webhook to the operator channel.

## Monitoring and alerts

- Coolify healthchecks restart unhealthy containers.
- Uptime probe (external, every 5 minutes) on the three health URLs.
- Safety pipeline: alert when `FLAGGED` rate exceeds 20 % of submissions in an
  hour or the classifier budget passes 80 %.
- Economy: alert on any `Payout` in `FAILED` for more than one hour.

## Scheduled jobs (Coolify scheduled tasks)

| Job | Command | Cadence |
|---|---|---|
| Weekly digest | `curl -X POST -H "Authorization: Bearer $PLATFORM_ADMIN_TOKEN" https://api.adapptive.space/api/platform-admin/digests/run` | Monday 09:00 UTC |
| Payout batch (when enabled) | `POST /api/platform-admin/payouts/run` | Friday 12:00 UTC |
| Backup | `deploy/backup.sh` | daily |

`PLATFORM_ADMIN_TOKEN` is an Authentik service-account token whose email is in
`PLATFORM_ADMIN_EMAILS`.

## Incident response

1. **Detect**: uptime probe, GlitchTip alert, or user report to
   `security.txt` contact.
2. **Triage** (15 min): which service, since when, blast radius (one tenant,
   all tenants, agents).
3. **Contain**: Coolify → redeploy previous image; for credential leaks rotate
   the affected key kind (`Settings → rotate`, or `IP_HASH_SALT` /
   `STUDIO_TOKEN_SECRET` in Coolify for platform secrets); for safety-pipeline
   bypass set `SAFETY_HARD_FLAG_ALL=true` so every new item is FLAGGED until
   fixed.
4. **Recover**: verify health table; run the golden-path E2E.
5. **Learn**: blameless write-up in `docs/incidents/YYYY-MM-DD-<slug>.md`
   within 5 working days; add a regression test.

## Release management

- Extension: `pnpm --filter @adapptive/extension package` → zip under
  `tools/browser-extension/release/`; the dashboard `/download` serves the
  latest. Chrome Web Store submission uses `STORE_LISTING.md` and `PRIVACY.md`
  in the extension folder; bump `manifest.json` version per submission.
- SDK: tag `sdk-v*` triggers `.github/workflows/sdk-publish.yml` (npm
  `NPM_TOKEN` secret required in the repository settings).
- MCP server: versioned with the repo; the remote service redeploys with the
  stack. Registry listing: `tools/mcp-server/server.json` is the MCP registry
  manifest; publish with `mcp-publisher login github && mcp-publisher publish`
  from that folder after each version bump (operator step; the GitHub
  namespace `io.github.DrakeN1721/*` is authorised by the repository owner's
  login).

## Access and secrets

- Coolify UI: loopback + SSH tunnel over Tailscale only.
- All secrets in Coolify's env store; never in git, never in images.
- Operator accounts: Authentik with TOTP/WebAuthn enforced; `akadmin`
  deactivated after bootstrap.