Operations

Runbooks for the people who run ADapptive. Deployment mechanics are in deploy/README.md; this page covers everything after "it is deployed".

Production map

Coolify auto-deploys main. CI (.github/workflows/ci.yml) must be green before merging to main.

Deploy checklist

1. CI green on the PR.
2. Migrations are additive and reviewed; destructive changes need a two-step plan (deploy code that tolerates both shapes, then migrate).
3. New env vars added to Coolify before the deploy (see the variable table in deploy/README.md; new since the level-up: SAFETY_CLASSIFIER, SAFETY_DAILY_BUDGET_USD, TRUSTED_ACCOUNT_MIN_AGE_HOURS, MCP_PUBLIC_URL, X402_*, HOUSE_POOL_AUTO_APPROVE, SENTRY_DSN, and the paid-campaign Stripe variables documented below).
4. After deploy: probe the health table above; open the dashboard; run one get_ready_feedback against the remote MCP with a test key.

Apex domain (one-time operator step)

In Coolify → the dashboard service → Domains: https://adapptive.space,https://app.adapptive.space. Traefik provisions certificates for both; the Next.js host-based redirect sends apex traffic to app.adapptive.space with a 308.

Authentik (operator steps)

• Dashboard client and extension public client: deploy/README.md.
• Tester enrollment: create an enrollment flow bound to the tester application slug (adapptive-testers), with email verification and a captcha stage. Set OIDC_AUDIENCE to include its client id. Leave disabled for closed beta.
• Rotate the signing key yearly; the API reads JWKS at runtime.

Current operator checkpoint — September 14, 2026

Operator-host health checks run every five minutes, and database backups run daily at 03:00 UTC. Each backup is restored into a fresh isolated PG17 clone. These user services are installed on the operator Linux host, separate from HS2; they require that machine to stay online. The repository's deploy/operator-monitoring.md documents installation and evidence. Database-only backups are active; MinIO backup, Sentry/GlitchTip and delivered external alerts remain open. Failed checks appear in systemd/journald. The restic configuration below remains an alternative runbook, not a claim that restic is scheduled in this deployment.

Backups

• Postgres: deploy/backup.sh runs pg_dump --format=custom and pushes to a restic repository off-host; retention --keep-daily 7 --keep-weekly 5 --keep-monthly 12. Schedule it as a Coolify scheduled task (daily 03:00 UTC) on the postgres service or a sidecar with the restic binary.
• MinIO: mc mirror the adapptive-screenshots bucket into the same restic repository weekly, or enable bucket versioning plus lifecycle rules.
• Restore drill (quarterly): restore the latest dump into adapptive_restore_test, run pnpm --filter @adapptive/api test against it with SEED_RESET disabled, record the time-to-restore.

Error tracking

GlitchTip (Sentry-compatible) can run as its own Coolify resource (deploy/docker-compose.glitchtip.yml). Set SENTRY_DSN on the API, dashboard, and MCP services; the SDKs are initialised only when the DSN is present. Alert routing: GlitchTip → email/webhook to the operator channel.

Monitoring and alerts

• Coolify healthchecks restart unhealthy containers.
• Uptime probe (external, every 5 minutes) on the three health URLs.
• Safety pipeline: alert when FLAGGED rate exceeds 20 % of submissions in an hour or the classifier budget passes 80 %.
• Economy: alert on any paid round in PAYMENT_DISPUTED, any payout attempt in UNKNOWN for more than one hour, any returned payment, or a ledger/reconciliation difference.

Paid campaign operations

Configure account-specific ADAPPTIVE_STRIPE_ACCOUNT_ID=acct_1UCbw2LxZ9fM8ShN, ADAPPTIVE_STRIPE_MODE, separate ADAPPTIVE_STRIPE_BILLING_KEY and restricted ADAPPTIVE_STRIPE_PAYOUT_KEY, both account-specific webhook secrets, the funded ADAPPTIVE_PAYOUT_FINANCIAL_ACCOUNT, and DASHBOARD_PUBLIC_URL. Generic Stripe keys are not used. Keep PAID_CAMPAIGNS_LIVE_APPROVED=false for test mode. Before setting it to true, record business/provider approval and publish the participation, payment, dispute, cancellation, privacy, and refund terms.

Register Stripe webhook delivery for Checkout completion, asynchronous payment success/failure, session expiry, refunds, disputes, subscription/invoice changes, and Identity session changes. The billing endpoint is POST /api/payments/stripe/webhook; it needs the unmodified request body.

Use a separate account-specific secret and event destination at POST /api/payments/stripe/payout-webhook for recipient/outbound-payment events. The operator page /admin/paid manages organization verification, founding launch, custom scopes, response reviews, QA delegation and manually approved fiat batches. Historical weighted quote, retest and manual-reconciliation controls remain separate. A reported or unknown payment is never eligible for resend. Only an authoritative failed, canceled, or returned state opens the retry path.

The dedicated fiat-worker is disabled by default. Enable ADAPPTIVE_FIAT_WORKER_ENABLED only after account/sandbox/funding checks. It runs checkout expiration and reconciliation of existing transfers only. It never submits payments. Operators prepare a draft batch, then explicitly approve each payment's amount, destination and maximum total debit; the API records their identity. Monitor READY holds, UNKNOWN/SUBMITTING payments, review deadlines, expiring corridor evidence and available funding. SMTP additionally requires DEVELOPER_EMAIL_ENABLED, a verified sender and eligible consent. Never retry UNKNOWN or stranded SENDING email without checking transport logs. See paid campaigns for current terms.

Run the developer-owned worker from the customer's repository host, following tools/campaign-worker/README.md. Monitor that process separately from the ADapptive control plane. Its MCP key must be restricted to the configured Project.

Scheduled jobs (Coolify scheduled tasks)

PLATFORM_ADMIN_TOKEN is an Authentik service-account token whose email is in PLATFORM_ADMIN_EMAILS.

Incident response

1. Detect: uptime probe, GlitchTip alert, or user report to security.txt contact.
2. Triage (15 min): which service, since when, blast radius (one tenant, all tenants, agents).
3. Contain: Coolify → redeploy previous image; for credential leaks rotate the affected key kind (Settings → rotate, or IP_HASH_SALT / STUDIO_TOKEN_SECRET in Coolify for platform secrets); for safety-pipeline bypass set SAFETY_HARD_FLAG_ALL=true so every new item is FLAGGED until fixed.
4. Recover: verify health table; run the golden-path E2E.
5. Learn: blameless write-up in docs/incidents/YYYY-MM-DD-<slug>.md within 5 working days; add a regression test.

Release management

• Extension: pnpm --filter @adapptive/extension package → zip under tools/browser-extension/release/; the dashboard /download serves the latest. Chrome Web Store submission uses STORE_LISTING.md and PRIVACY.md in the extension folder; bump manifest.json version per submission.
• SDK: tag sdk-v* triggers .github/workflows/sdk-publish.yml (npm NPM_TOKEN secret required in the repository settings).
• MCP server: versioned with the repo; the remote service redeploys with the stack. Registry listing: tools/mcp-server/server.json is the MCP registry manifest; publish with mcp-publisher login github && mcp-publisher publish from that folder after each version bump (operator step; the GitHub namespace io.github.DrakeN1721/* is authorised by the repository owner's login).

Access and secrets

• Coolify UI: loopback + SSH tunnel over Tailscale only.
• All secrets in Coolify's env store; never in git, never in images.
• Operator accounts: Authentik with TOTP/WebAuthn enforced; akadmin deactivated after bootstrap.