Operations

Runbooks for the people who run ADapptive. Deployment mechanics are in deploy/README.md; this page covers everything after "it is deployed".

Production map

Coolify auto-deploys main. CI (.github/workflows/ci.yml) must be green before merging to main.

Deploy checklist

1. CI green on the PR.
2. Migrations are additive and reviewed; destructive changes need a two-step plan (deploy code that tolerates both shapes, then migrate).
3. New env vars added to Coolify before the deploy (see the variable table in deploy/README.md; new since the level-up: SAFETY_CLASSIFIER, SAFETY_DAILY_BUDGET_USD, TRUSTED_ACCOUNT_MIN_AGE_HOURS, MCP_PUBLIC_URL, X402_*, HOUSE_POOL_AUTO_APPROVE, SENTRY_DSN).
4. After deploy: probe the health table above; open the dashboard; run one get_ready_feedback against the remote MCP with a test key.

Apex domain (one-time operator step)

In Coolify → the dashboard service → Domains: https://adapptive.space,https://app.adapptive.space. Traefik provisions certificates for both; the Next.js host-based redirect sends apex traffic to app.adapptive.space with a 308.

Authentik (operator steps)

• Dashboard client and extension public client: deploy/README.md.
• Tester enrollment: create an enrollment flow bound to the tester application slug (adapptive-testers), with email verification and a captcha stage. Set OIDC_AUDIENCE to include its client id. Leave disabled for closed beta.
• Rotate the signing key yearly; the API reads JWKS at runtime.

Backups

• Postgres: deploy/backup.sh runs pg_dump --format=custom and pushes to a restic repository off-host; retention --keep-daily 7 --keep-weekly 5 --keep-monthly 12. Schedule it as a Coolify scheduled task (daily 03:00 UTC) on the postgres service or a sidecar with the restic binary.
• MinIO: mc mirror the adapptive-screenshots bucket into the same restic repository weekly, or enable bucket versioning plus lifecycle rules.
• Restore drill (quarterly): restore the latest dump into adapptive_restore_test, run pnpm --filter @adapptive/api test against it with SEED_RESET disabled, record the time-to-restore.

Error tracking

GlitchTip (Sentry-compatible) runs as its own Coolify resource (deploy/docker-compose.glitchtip.yml). Set SENTRY_DSN on the API, dashboard, and MCP services; the SDKs are initialised only when the DSN is present. Alert routing: GlitchTip → email/webhook to the operator channel.

Monitoring and alerts

• Coolify healthchecks restart unhealthy containers.
• Uptime probe (external, every 5 minutes) on the three health URLs.
• Safety pipeline: alert when FLAGGED rate exceeds 20 % of submissions in an hour or the classifier budget passes 80 %.
• Economy: alert on any Payout in FAILED for more than one hour.

Scheduled jobs (Coolify scheduled tasks)

PLATFORM_ADMIN_TOKEN is an Authentik service-account token whose email is in PLATFORM_ADMIN_EMAILS.

Incident response

1. Detect: uptime probe, GlitchTip alert, or user report to security.txt contact.
2. Triage (15 min): which service, since when, blast radius (one tenant, all tenants, agents).
3. Contain: Coolify → redeploy previous image; for credential leaks rotate the affected key kind (Settings → rotate, or IP_HASH_SALT / STUDIO_TOKEN_SECRET in Coolify for platform secrets); for safety-pipeline bypass set SAFETY_HARD_FLAG_ALL=true so every new item is FLAGGED until fixed.
4. Recover: verify health table; run the golden-path E2E.
5. Learn: blameless write-up in docs/incidents/YYYY-MM-DD-<slug>.md within 5 working days; add a regression test.

Release management

• Extension: pnpm --filter @adapptive/extension package → zip under tools/browser-extension/release/; the dashboard /download serves the latest. Chrome Web Store submission uses STORE_LISTING.md and PRIVACY.md in the extension folder; bump manifest.json version per submission.
• SDK: tag sdk-v* triggers .github/workflows/sdk-publish.yml (npm NPM_TOKEN secret required in the repository settings).
• MCP server: versioned with the repo; the remote service redeploys with the stack. Registry listing: tools/mcp-server/server.json is the MCP registry manifest; publish with mcp-publisher login github && mcp-publisher publish from that folder after each version bump (operator step; the GitHub namespace io.github.DrakeN1721/* is authorised by the repository owner's login).

Access and secrets

• Coolify UI: loopback + SSH tunnel over Tailscale only.
• All secrets in Coolify's env store; never in git, never in images.
• Operator accounts: Authentik with TOTP/WebAuthn enforced; akadmin deactivated after bootstrap.