SOP-004: E2E Testing

SOP-004: E2E Testing

Purpose

Run and interpret end-to-end tests against services deployed on staging. E2E tests validate the full stack: HTTP endpoints, database state, authentication, multi-tenancy isolation, and inter-service communication.

Scope

Applies to all services deployed to staging with a health check endpoint. E2E tests run AFTER successful deployment (SOP-003), never against local development instances.

Prerequisites

Procedure

1. Verify staging is healthy

FQDN="https://{service}.staging.orbusdigital.com"
HEALTH="/health"  # /api/health for Node services
curl -sf "${FQDN}${HEALTH}" | python3 -m json.tool

If unhealthy, do not proceed – follow SOP-005 (Incident Response).

2. Generate E2E scenarios (if not exists)

Spawn scenario agent:

/agent scenario "SERVICE: {service}. PROJECT: {project}. Generate API + browser E2E scenarios. Probe staging with curl before writing. Write to ~/dev/projects/{service}/tests/e2e/"

Critical rules for scenario generation (from lessons learned): - Probe staging first: curl every route before writing scenarios to use concrete URLs, not template variables - No template variables: {{SLUG}} style placeholders are forbidden in final output (lesson from 2026-03-23) - Valid JSON only: No // comments in JSON files (lesson from 2026-03-23) - Status fields: Use RUNNABLE, BLOCKED_EXTERNAL, or FAIL_EXPECTED – never leave blank - Validate: python3 -m json.tool < e2e-scenarios.json must succeed

3. Run E2E tests

Spawn E2E test agent:

/agent e2e-test "SERVICE: {service}. PROJECT: {project}. Execute API + browser E2E tests against staging URL ${FQDN}. Scenarios: ~/dev/projects/{service}/tests/e2e/"

The E2E agent will: - Execute each RUNNABLE scenario via curl (API tests) or PinchTab (browser tests) - Record HTTP status codes, response bodies, timing - Compare against expected outcomes in the scenario file - Write results to ~/dev/ops/reviews/{service}/e2e-report.json

4. For Rust services: pre-build before E2E

If running E2E against a local instance (rare), always pre-build:

cd ~/dev/projects/{service}
cargo build --release

Then start the binary directly from target/release/, not via cargo run (lesson from notification-hub: compilation eats the timeout).

5. Interpret results

Read the E2E report:

cat ~/dev/ops/reviews/{service}/e2e-report.json | python3 -c "
import sys, json
r = json.load(sys.stdin)
print(f'Verdict: {r[\"verdict\"]}')
print(f'API tests: {len([t for t in r.get(\"apiTests\",[]) if t[\"status\"]==\"PASS\"])} pass')
print(f'Failures: {len([t for t in r.get(\"apiTests\",[]) if t[\"status\"]==\"FAIL\"])}')
"

Verdict interpretation: | Verdict | Meaning | Action | |———|———|——–| | E2E_PASS | All RUNNABLE scenarios passed | Proceed to staging-verified state | | E2E_FAIL | One or more scenarios failed | Analyze failures, spawn dev fix | | E2E_BLOCKED | Cannot run (infra issue, missing deps) | Follow SOP-007 |

Common failure patterns: - 401/403 on authenticated endpoints: check JWT validation, tenant_id extraction - 500 on database queries: check schema migration ran, search_path set - Cross-tenant data leak (CRITICAL): RLS policy missing or bypassed – immediate security escalation - Empty response body from proxy: strip content-encoding headers (lesson from 2026-03-22)

6. Write status

CLI="$HOME/dev/ops/adlc-v2/scripts/cli"
bash $CLI/write-status.sh {service} e2e-test DONE "X/Y scenarios passed"
bash $CLI/write-pipeline-state.sh {project} {service} E2E_PASS "X/Y pass"

Or on failure:

bash $CLI/write-status.sh {service} e2e-test FAILED "X failures -- see e2e-report.json"
bash $CLI/write-pipeline-state.sh {project} {service} E2E_FAIL "X failures"

Verification

Rollback

E2E tests are read-only against staging (they should not mutate persistent state in a way that affects other tests). If E2E tests created test data: - Test data should use a dedicated test tenant (tenant_id = "e2e-test-tenant") - Clean up: DELETE FROM {table} WHERE tenant_id = 'e2e-test-tenant'

If E2E tests exposed a critical bug in staging: - Follow SOP-005 (Incident Response) for the staging service - Do not re-run E2E until the fix is deployed

References