ADR-006: Out-of-Band Status Page and Incident Management Topology

Status

ACCEPTED  2026-09-12

Context

In an enterprise healthcare SaaS monorepo serving clinics, compounding pharmacies, doctors, and patients, system availability and operational transparency are paramount. Unscheduled downtime, network partitioning, or infrastructure maintenance must be communicated transparently to preserve trust and prevent operational paralysis across partner compounding stations and clinics.

The Pitfalls of Co-Located Status Pages

1. Circular Dependency & Fate-Sharing Anti-Pattern:

   • In naive architectures, status pages or monitoring dashboards are hosted within the primary application cluster or the same cloud provider/region (e.g. running inside containers in the primary cloud environment).
   • If the primary cloud infrastructure experiences regional network degradation, control plane failure, or regional outage in São Paulo (southamerica-east1), the status page inevitably crashes along with the services it is supposed to report on.
   • Users encountering connection timeouts receive raw browser DNS or connection-refused errors without any status updates, overwhelming customer support channels.

2. Telemetry Poisoning from Internal Monitors:

   • Monitoring probes that execute from within the same internal private VPC or serverless cluster fail to detect external ingress failures, Cloudflare Tunnel drops, DNS hijacking, SSL certificate expirations, or regional ISP routing partitions.
   • Reliable observability requires synthetic probes executing from external vantage points across diverse geographical continents.

3. Ad-Hoc Incident Communication:

   • Without a formalized incident state machine, incident reporting degrades into unstructured chat messages, ambiguous status emails, or uncommunicated silent mitigations, complicating post-incident regulatory audits (CFM Res. 1.821, ANVISA RDC 67/2007).

Decision

The GERCIA platform establishes a 100% Out-of-Band Public Status Page and Incident Management Topology hosted at status.gercia.com.br, decoupled from the primary cloud infrastructure (Google Cloud Platform and Cloudflare Pages):

flowchart TD
    subgraph GLOBAL_USERS ["External Stakeholders"]
        DOCTORS["Doctors & Clinics"]
        PHARMACIES["Compounding Pharmacies"]
        PATIENTS["Patients (Mobile Client)"]
        SRE["GERCIA DevSecOps Team"]
    end

    subgraph OUT_OF_BAND ["Cloudflare Anycast Global Edge (Out-of-Band Architecture)"]
        STATUS_PAGE["status.gercia.com.br<br>(Astro 7 + Tailwind v4 + Svelte 5)<br>Cloudflare Pages: gercia-status"]
        PAGES_FN["Pages Function (/api/status)<br>Edge Runtime with D1 Binding"]
        D1_DB[("Cloudflare D1 Serverless<br>(gercia-status-prd-db)<br>Globally Replicated SQLite")]
        CF_HEALTH["Cloudflare Distributed Health Checks<br>(Probing from SSAM, WNAM, WEU every 60s)"]
        CF_ALERTS["Cloudflare Alerting Engine v3<br>(health_check_status_notification)"]
    end

    subgraph PRIMARY_INFRA ["Primary Cloud Infrastructure (GCP southamerica-east1 & Cloudflare)"]
        subgraph TUNNELS ["Cloudflare Ingress Tunnels"]
            T_PRD["gercia-tunnel-svc-prd"]
            T_STG["gercia-tunnel-svc-stg"]
        end

        subgraph SERVICES ["Production Workloads"]
            SERVER["Gercia.Server (Cloud Run • api.gercia.com.br/health)"]
            WEB["@gercia/app (Cloudflare Pages • app.gercia.com.br)"]
            SITE["@gercia/site (Cloudflare Pages • gercia.com.br)"]
            DOCS["Public Help Center (Cloudflare Pages • docs.gercia.com.br)"]
            EDGE["Hardware Station Hubs (/hubs/*)"]
        end
    end

    subgraph NOTIFICATION_CHANNELS ["Incident & SRE Communication"]
        GCHAT["Google Chat Incidents Space<br>(WEBHOOK_GERCIA_INCIDENTS)"]
        EMAIL["SRE Alerting Dispatch<br>(alerts@gercia.com.br)"]
    end

    DOCTORS & PHARMACIES & PATIENTS --> STATUS_PAGE
    STATUS_PAGE -- "Client Hydration & SWR (every 30s)" --> PAGES_FN
    PAGES_FN -- "Native SQL Query" --> D1_DB

    CF_HEALTH -- "HTTPS GET /health (60s)" --> SERVER
    CF_HEALTH -- "HTTPS GET / (60s)" --> WEB
    CF_HEALTH -- "HTTPS GET / (60s)" --> SITE
    CF_HEALTH -- "HTTPS GET / (60s)" --> DOCS

    CF_HEALTH -- "On Failure / Degradation" --> CF_ALERTS
    CF_ALERTS --> EMAIL
    CF_ALERTS --> GCHAT

    SRE -- "D1 SQL / API Mutations" --> D1_DB

1. Architectural Principles & Isolation

• Complete Datacenter Decoupling: The status application resides in apps/status/ and is built using Astro 7, Tailwind CSS v4, and Svelte 5, deployed directly to Cloudflare Pages (gercia-status.pages.dev). Even in the event of a catastrophic primary regional cloud outage (such as GCP southamerica-east1), status.gercia.com.br remains 100% operational across Cloudflare’s Anycast network (330+ edge PoPs globally).
• Cloudflare D1 Serverless Database & Environment Segregation: Incident history, component statuses, and operational logs are persisted in dedicated Cloudflare D1 serverless databases completely decoupled from the PostgreSQL database on Google Cloud SQL. Production (status.gercia.com.br) binds to gercia-status-prd-db, while Staging previews bind to gercia-status-stg-db, preventing test simulations, drills, or schema migrations from polluting the public production status page. Dynamic queries are executed at the edge through Cloudflare Pages Functions (/api/status) with edge caching (Cache-Control: public, max-age=15, stale-while-revalidate=60).
• Universal SSL Single-Level Compliance: Per monorepo subdomain governance (Network Topology), status.gercia.com.br is a single-level subdomain fully covered by Cloudflare Universal SSL certificates without requiring multi-level certificate purchases.
• Dual-Channel Probe Architecture:
  1. External Synthetic Edge Probes: Cloudflare Health Checks (POST /zones/{zone_id}/healthchecks) querying production endpoints from Southern South America (SSAM), Western North America (WNAM), and Western Europe (WEU) every 60 seconds with 2 consecutive retry requirements before declaring degradation.
  2. Reactive Client Hydration: The status front-end pre-renders static HTML with instantaneous paint, while a reactive Svelte component periodically re-validates against /api/status every 30 seconds and provides manual refresh capability.
• Dynamic 90-Day Rolling Availability & Maintenance Modeling: The status visualizer calculates availability dynamically across a rolling 90-day window ($90 \times 24 \times 60 = 129,600$ minutes). Monitored components render 90 interactive daily blocks with distinct visual states: Operational (Green), Scheduled Maintenance (Blue), Degraded (Amber), and Outage (Rose). Planned maintenance windows (e.g., database upgrades, security hardening) are visually segregated from unscheduled downtime, preserving historical transparency while aligning with industry standards where scheduled maintenance is contractually excluded from monthly billing penalties.

2. Monitored Component & Health Check Taxonomy

Active synthetic probes are named consistently with monorepo project identifiers:

Probe Name	Monitored Address	Architectural Scope & Target	Probe Configuration
gercia-api	api.gercia.com.br	REST API Gateway, Sovereign OpenIddict STS, and AI MCP (Gercia.Server).	HTTPS GET /health (HTTP 200, 60s, SSAM/WNAM/WEU)
gercia-app	app.gercia.com.br	Multi-Tenant Enterprise Client (@gercia/app) for Doctors, Clinics, and Compounding Pharmacies.	HTTPS GET / (HTTP 200, 60s, SSAM/WNAM/WEU)
gercia-site	gercia.com.br	Public institutional website (Astro on Cloudflare Pages).	HTTPS GET / (HTTP 200, 60s, SSAM/WNAM/WEU)
gercia-docs	docs.gercia.com.br	Public Help Center and User Manuals (Astro Starlight).	HTTPS GET / (HTTP 200, 60s, SSAM/WNAM/WEU)

Zero Trust Access Exclusions

Internal portals protected behind Cloudflare Zero Trust with mandatory Google Workspace authentication (ops.gercia.com.br) and private engineering documentation (eng.gercia.com.br) are deliberately excluded from public health check probes to prevent spurious false-positive alerts triggered by 302 OIDC redirect challenges.

3. Standardized 4-Stage SRE Incident Lifecycle

Operational incident response follows a strict 4-stage progression:

flowchart LR
    STAGE1["1. Investigating<br>SLA: ≤ 30 min (P1)<br>Triaging alert / probe failures"] --> STAGE2["2. Identified<br>Root cause isolated<br>Mitigation in active progress"]
    STAGE2 --> STAGE3["3. Monitoring<br>Fix applied to servers<br>Verifying probe stabilization"]
    STAGE3 --> STAGE4["4. Resolved<br>SLA: ≤ 4 hours (P1)<br>100% nominal & post-mortem"]

1. Investigating (Investigando): Initial detection of probe degradation or telemetry anomaly. SRE on-call acknowledges alert, inserts incident in D1 (incidents and incident_updates), and notifies the #gercia-incidents Google Chat space.
2. Identified (Identificado): Root cause isolated (e.g., transit route degradation, database connection pool exhaustion, memory pressure). Remediation plan enacted.
3. Monitoring (Monitorando): Code fix or infrastructure mitigation applied. SRE verifies synthetic probe stabilization and error rate drop over a 15-minute observation window.
4. Resolved (Resolvido): All synthetic probes green, services fully operational, and incident closed with post-mortem documentation.

Public Incident Transparency & Sanitization Policy

All public-facing incident titles and operational progress messages must be written with professional clarity focusing on service impact, recovery steps, and customer reassurance. They must strictly omit internal infrastructure hostnames, vendor partner names (e.g., bare-metal server providers), OS kernel distributions, and specific database engine versions to protect system security and prevent internal architectural leakage.

4. Automated CI/CD Deployment

The status page is built and deployed via deploy/cloudbuild.status.yaml using Google Cloud Build:

npm ci && npm run build
npx wrangler pages deploy dist --project-name="gercia-status" --branch="$BRANCH_NAME" --commit-hash="$COMMIT_SHA"

Automatic deployment notifications dispatch directly to the Google Chat DevSecOps space (WEBHOOK_GERCIA_DEVSECOPS).

Consequences

Positive

• 100% Fault Tolerance: Complete regional outage on primary cloud infrastructure does not affect the public availability of status.gercia.com.br.
• Proactive Transparency: External synthetic probes detect issues from multiple continents before users report them.
• Serverless D1 Persistence: Incidents and updates persist in Cloudflare D1 without maintaining or paying for a separate database cluster.
• Modern Astro + Tailwind Stack: Codebase mirrors apps/site/ standards with high developer velocity and sub-second compilation.
• Zero Host Overhead: Static edge execution and edge functions consume zero memory or CPU on the primary cloud infrastructure.

Trade-offs & Mitigations

• Secondary DNS & Deployment Management: Requires maintaining Cloudflare Pages configurations, D1 bindings, and health check policies. Mitigated through monorepo code tracking in deploy/cloudbuild.status.yaml and programmatic MCP automation.
• Zero Trust Access Exclusions: Managed backoffice endpoints (ops.gercia.com.br) rely on internal telemetry rather than public edge probes to avoid authentication redirect collisions.