SOP-003: Staging Deployment via Coolify

SOP-003: Staging Deployment via Coolify

Purpose

Deploy a service to the staging environment on srv-staging (35.195.54.220) using Coolify. Covers pre-deploy checks, Coolify API interaction, health verification, and post-deploy validation.

Scope

Applies to all ODS Platform services that run as long-lived HTTP servers. Does not apply to CLI tools (e.g., migration scripts) or library crates (e.g., ods-common). For non-HTTP services, see SOP-007 (External Blockers) for escalation.

Prerequisites

Procedure

1. Source credentials and read Coolify config

source ~/.env.adlc
SERVICE="{service}"
UUID=$(python3 -c "import json; print(json.load(open('$HOME/dev/ops/coolify/$SERVICE.json'))['coolify']['app_uuid'])")
echo "Deploying $SERVICE (UUID: $UUID)"

2. Verify the service is configured in Coolify

curl -sf "$COOLIFY_API_URL/api/v1/applications/$UUID" \
  -H "Authorization: Bearer $COOLIFY_API_TOKEN" | python3 -m json.tool | head -20

If 404: the app needs to be created in Coolify UI first. Post to Slack DM:

:eyes: HUMAN REVIEW -- {service} needs first-time Coolify setup. Create app in Coolify UI, save UUID to ~/dev/ops/coolify/{service}.json

3. Ensure Docker network is set to coolify

Critical lesson (2026-03-21): containers must be on the coolify network to reach ods-postgres and redpanda.

curl -sf "$COOLIFY_API_URL/api/v1/applications/$UUID" \
  -H "Authorization: Bearer $COOLIFY_API_TOKEN" | python3 -c "
import sys, json
d = json.load(sys.stdin)
net = d.get('docker_network', d.get('settings', {}).get('docker_network', 'unknown'))
print(f'Docker network: {net}')
if net != 'coolify':
    print('WARNING: Network must be coolify!')
"

4. Trigger deployment (restart = full rebuild)

curl -sf -X POST "$COOLIFY_API_URL/api/v1/applications/$UUID/restart" \
  -H "Authorization: Bearer $COOLIFY_API_TOKEN" \
  -H "Content-Type: application/json"

Note: Use /restart not /deploy or /redeploy – this Coolify version only supports /restart (lesson from 2026-03-22).

5. Poll health check

For Rust services, use a 6-minute window (36 polls x 10s) due to long compile times (lesson from 2026-03-23):

FQDN=$(python3 -c "import json; print(json.load(open('$HOME/dev/ops/coolify/$SERVICE.json'))['coolify']['fqdn'])")
HEALTH_PATH="/health"  # Use /api/health for Node services

MAX_POLLS=36  # 6 minutes for Rust, 18 for Node
for i in $(seq 1 $MAX_POLLS); do
  code=$(curl -sf -o /dev/null -w "%{http_code}" "${FQDN}${HEALTH_PATH}" 2>/dev/null || echo "000")
  echo "Poll $i/$MAX_POLLS: HTTP $code"
  [ "$code" = "200" ] && echo "HEALTHY" && break
  sleep 10
done

6. Verify deployment

# Full health check
curl -sf "${FQDN}${HEALTH_PATH}" | python3 -m json.tool

# Check response headers
curl -sI "${FQDN}${HEALTH_PATH}" | head -10

7. Write status and pipeline state

CLI="$HOME/dev/ops/adlc-v2/scripts/cli"
bash $CLI/write-status.sh $SERVICE deploy DEPLOYED "Deployed to $FQDN"
bash $CLI/write-pipeline-state.sh {project} $SERVICE STAGING_DEPLOYED "$FQDN"

8. Post milestone to Slack

source ~/.env.adlc
curl -sf -X POST "https://slack.com/api/chat.postMessage" \
  -H "Authorization: Bearer $SLACK_BOT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$(python3 -c "import json; print(json.dumps({'channel':'C0AN0N8AUGZ','text':':white_check_mark: $SERVICE deployed to staging -- $FQDN'}))")"

Verification

Rollback

If deployment fails or service is unhealthy:

  1. Check Coolify build logs via API:
curl -sf "$COOLIFY_API_URL/api/v1/applications/$UUID/logs" \
  -H "Authorization: Bearer $COOLIFY_API_TOKEN" | tail -50
  1. If the issue is in new code, revert to previous commit on staging branch and redeploy:
cd ~/dev/projects/$SERVICE
git log --oneline -5  # Find last known good commit
# Revert on staging branch, push, then restart via Coolify
  1. If infrastructure issue (network, DNS, TLS): escalate to Slack DM as BLOCKED.

References