SRE Incident Management & Status Page Runbook

This operational runbook defines the standardized procedures, communication protocols, and tooling commands for GERCIA Site Reliability Engineers (SREs) and on-call developers during service degradation, unexpected outages, or planned maintenance windows.

Out-of-Band Status Page Location

The official, datacenter-decoupled public status page is accessible 24/7 at:

➡️ https://status.gercia.com.br  (Fallback mirror: https://gercia-status.pages.dev)

---

1. Incident Severity Matrix & Resolution SLAs

Whenever an anomaly is reported or detected by automated Cloudflare Health Checks, the on-call engineer must categorize the incident within 15 minutes according to the platform SLA matrix (ADR-006):

Severity Tier	Response SLA	Target Resolution SLA	Escalation Triggers & Operational Scope
🔴 P1 - Urgent	≤ 30 min	≤ 4 hours	Total Outage / Safety Threat: Doctors unable to issue digital prescriptions, compounding pharmacies blocked from balance weighing, REST API completely down, or total database outage.
🟠 P2 - High	≤ 2 hours	≤ 48 hours	Core Service Degradation: Elevated API latency (> 1500ms), partial mobile push notification delivery failure, or Staging environment pipeline failure blocking release verification.
🟡 P3 - Medium	≤ 8 hours	Current Sprint	Non-Critical Issue: Single background job failure with automated retry, cosmetic portal glitch, or non-blocking documentation portal latency.
🔵 P4 - Low	≤ 24 hours	Next Milestone	Minor Technical Debt: Internal telemetry warning, log shipper transient buffer warning, or scheduled certificate renewal reminder.

---

2. Availability Calculation Standards: Contractual SLA vs. Status Page Observability

The GERCIA platform maintains an authoritative separation between Contractual Monthly SLAs (B2B legal terms) and Public Status Page Observability (SRE transparency):

flowchart TD
    subgraph DUAL_PERIMETER ["Availability & Uptime Models"]
        subgraph SLA_LEGAL ["1. Contractual SLA (Legal / B2B Contracts)"]
            SLA_WINDOW["Window: Calendar Month (1st to last day)<br>• Aligned to monthly billing cycle"]
            SLA_EXCLUSION["Excluded Downtime Clause:<br>• Scheduled preventive maintenance with 48h advance notice<br>is EXCLUDED from downtime penalties"]
            SLA_PENALTY["Financial Remediation:<br>• Service Credits (10% to 50% discount on next invoice)"]
        end

        subgraph STATUS_OBS ["2. Public Status Page (Real-Time Transparency)"]
            STATUS_WINDOW["Window: Rolling 90 Days (Today - 89 days)<br>• Industry standard (GitHub, Cloudflare, OpenAI)"]
            STATUS_METRIC["Full Observability:<br>• 100% of events recorded<br>• Scheduled Maintenance rendered in Dedicated Blue (#3b82f6)"]
            STATUS_BENEFIT["Brand Value:<br>• Technical transparency and customer confidence"]
        end
    end

Mathematical Uptime Formula (Rolling 90-Day Window)

On status.gercia.com.br, component availability is computed dynamically over the rolling 90-day window:

$$\text{Total Window Minutes} = 90 \times 24 \times 60 = 129,600\text{ minutes}$$

$$\text{Component Availability } (%) = \frac{129,600 - \sum \text{Downtime Minutes}}{129,600} \times 100$$

Daily Heatbar Visual States & Color Taxonomy

Each monitored component renders 90 interactive daily blocks with standardized status colors:

Status State	Heatbar Color	Trigger Criteria	Tooltip Format
Operational	bg-emerald-500 (Green)	Zero incidents or maintenance events on the given day.	[Date] • 100.0% operacional
Scheduled Maintenance	bg-blue-500 (Blue)	Planned maintenance window (e.g., kernel update, database upgrade).	[Date] • Manutenção Programada: [Title] ([Duration] min) • [Daily]% no dia
Degraded Performance	bg-amber-500 (Amber)	Elevated latency or minor non-blocking service impact.	[Date] • Degradação: [Title] ([Duration] min) • [Daily]% no dia
Partial Outage	bg-orange-500 (Orange)	High-severity incident or secondary component failure.	[Date] • Interrupção Parcial: [Title] ([Duration] min) • [Daily]% no dia
Major Outage	bg-rose-500 (Rose)	Critical failure blocking primary clinical or pharmacy operations.	[Date] • Interrupção Crítica: [Title] ([Duration] min) • [Daily]% no dia

SRE Incident Case Study: August 27, 2026 Maintenance

During the scheduled preventive maintenance on August 27, 2026 (INC-20260827-01, 06:00 to 06:45 UTC):

• Affected Workloads: gercia-app and gercia-api (Gercia.Server) (downtime: 45 minutes).
• Daily Availability on Aug 27: $\frac{1,440 - 45}{1,440} \times 100 = \mathbf{96.9%}$.
• 90-Day Rolling Uptime: $\frac{129,600 - 45}{129,600} \times 100 = \mathbf{99.97%}$ (well within the $99.9%$ Three Nines enterprise SLA target).
• Unaffected Services: gercia-site and gercia-docs maintained $\mathbf{100.0%}$ availability.
• Contractual Treatment: Because this window was scheduled and pre-announced during off-peak hours, it constitutes Excluded Downtime under GERCIA Master Service Agreements (MSAs), triggering zero service credit penalties while providing 100% public visibility on the status page.

---

3. Standardized 4-Stage SRE Lifecycle Playbook

flowchart TD
    DETECTION["Alert Detected<br>(Cloudflare Health Check / User Report)"] --> TRIAGE["1. Triage & Acknowledge<br>(Stage: Investigating)"]
    TRIAGE --> ISOLATION["2. Root Cause Analysis<br>(Stage: Identified)"]
    ISOLATION --> MITIGATION["3. Enact Mitigation & Hotfix<br>(Stage: Monitoring)"]
    MITIGATION --> RECOVERY["4. Verify Synthetic Probes & Close<br>(Stage: Resolved)"]
    RECOVERY --> POSTMORTEM["5. Blameless Post-Mortem<br>(docs/private/ SRE Archive)"]

Stage 1: Investigating (Investigando)

• Goal: Acknowledge the incident publicly, prevent support ticket flooding, and mobilize the incident commander.
• Actions:
  1. Acknowledge alert in the Google Chat #gercia-incidents space (WEBHOOK_GERCIA_INCIDENTS).
  2. Insert active incident into the Cloudflare D1 production database (gercia-status-prd-db):

     # Production Incident Execution (status.gercia.com.br)
     npx wrangler d1 execute gercia-status-prd-db --remote --command="INSERT INTO incidents (id, title, severity, status, impact, affected_components, started_at) VALUES ('INC-YYYYMMDD-01', '<Sanitized Title>', 'high', 'investigating', 'minor', '[\"gercia-app\", \"gercia-api\"]', datetime('now'));"
     npx wrangler d1 execute gercia-status-prd-db --remote --command="INSERT INTO incident_updates (id, incident_id, stage, message) VALUES ('UPD-01', 'INC-YYYYMMDD-01', 'Investigating', '<Sanitized Operational Message>');"
     
     # Staging SRE Drills & Test Simulation (staging.gercia-status.pages.dev)
     # Substitute database name with gercia-status-stg-db

  3. Edge caches automatically revalidate on the next request (SWR: 30s client hydration, 15s edge TTL).
  4. Dispatch the Initial Incident Card to #gercia-incidents.

Stage 2: Identified (Identificado)

• Goal: Pinpoint the technical failure mechanism and communicate the mitigation roadmap.
• Diagnostic Checklist:
  • Inspect Google Cloud SQL connections & active queries:

    gcloud sql instances describe gercia-db-prd --project=gercia-gcp

  • Verify Google Cloud Run service health:

    gcloud run services describe gercia-server-prd --project=gercia-gcp --region=southamerica-east1

  • Verify Cloudflare Tunnel status:

    cloudflared tunnel info gercia-tunnel-svc-prd

  • Insert progress update into Cloudflare D1 (incident_updates table):

    npx wrangler d1 execute gercia-status-prd-db --remote --command="INSERT INTO incident_updates (id, incident_id, stage, message) VALUES ('UPD-02', 'INC-YYYYMMDD-01', 'Identified', '<Sanitized Mitigation Details>');"

Stage 3: Monitoring (Monitorando)

• Goal: Apply the hotfix or failover procedure and observe telemetry stabilization.
• Actions:
  • Restart degraded container or execute automated database connection pool drain.
  • Monitor external synthetic probes on Cloudflare:

    # Synthetic probes live verification
    curl -I https://api.gercia.com.br/health
    curl -I https://app.gercia.com.br/health

  • Observe 15 minutes of green telemetry across South America (SSAM), North America (WNAM), and Europe (WEU).
  • Update incident status in Cloudflare D1:

    npx wrangler d1 execute gercia-status-prd-db --remote --command="UPDATE incidents SET status = 'monitoring' WHERE id = 'INC-YYYYMMDD-01';"
    npx wrangler d1 execute gercia-status-prd-db --remote --command="INSERT INTO incident_updates (id, incident_id, stage, message) VALUES ('UPD-03', 'INC-YYYYMMDD-01', 'Monitoring', 'Mitigation deployed. Verifying telemetry stabilization.');"

Stage 4: Resolved (Resolvido)

• Goal: Formally conclude the operational incident, confirm 100% nominal health, and trigger post-mortem review.
• Actions:
  • Conclude incident in Cloudflare D1:

    npx wrangler d1 execute gercia-status-prd-db --remote --command="UPDATE incidents SET status = 'resolved', resolved_at = datetime('now') WHERE id = 'INC-YYYYMMDD-01';"
    npx wrangler d1 execute gercia-status-prd-db --remote --command="INSERT INTO incident_updates (id, incident_id, stage, message) VALUES ('UPD-04', 'INC-YYYYMMDD-01', 'Resolved', 'Service fully restored and operating nominally.');"

  • Send the Incident Resolved Card to Google Chat (#gercia-incidents).
  • Open an issue in Linear (ENG team) for the Blameless Post-Mortem.

Public Incident Transparency & Sanitization Policy

All public-facing incident titles and operational progress messages must be written with professional clarity focusing on service impact, recovery steps, and customer reassurance. They must strictly omit internal infrastructure hostnames, vendor partner names (e.g., bare-metal server providers), OS kernel distributions, and specific database engine versions to protect system security and prevent internal architectural leakage.

---

4. Google Chat Incident Card Templates (WEBHOOK_GERCIA_INCIDENTS)

The automated sidecars and SRE scripts dispatch rich cards formatted for Google Chat spaces:

Incident Declaration Card (Stage 1: Investigating)

{
  "cardsV2": [
    {
      "cardId": "incident-investigating",
      "card": {
        "header": {
          "title": "🚨 INCIDENT DECLARED: GERCIA Production",
          "subtitle": "Severity: P1 Urgent | Investigating",
          "imageUrl": "https://status.gercia.com.br/favicon.svg",
          "imageType": "SQUARE"
        },
        "sections": [
          {
            "header": "Incident Details",
            "widgets": [
              {
                "textParagraph": {
                  "text": "<b>Impacted Services:</b> REST API Gateway (api.gercia.com.br)<br><b>Initial Detection:</b> Synthetic Probe Degradation in SSAM & WEU<br><b>Incident Commander:</b> @oncall-sre"
                }
              },
              {
                "buttonList": {
                  "buttons": [
                    {
                      "text": "View Status Page",
                      "onClick": {
                        "openLink": { "url": "https://status.gercia.com.br" }
                      }
                    },
                    {
                      "text": "Open Cloud SQL Insights",
                      "onClick": {
                        "openLink": { "url": "https://console.cloud.google.com/sql/instances/gercia-db-prd/insights" }
                      }
                    }
                  ]
                }
              }
            ]
          }
        ]
      }
    }
  ]
}

Incident Resolution Card (Stage 4: Resolved)

{
  "cardsV2": [
    {
      "cardId": "incident-resolved",
      "card": {
        "header": {
          "title": "🟢 INCIDENT RESOLVED: GERCIA Production",
          "subtitle": "All Systems 100% Operational",
          "imageUrl": "https://status.gercia.com.br/favicon.svg",
          "imageType": "SQUARE"
        },
        "sections": [
          {
            "header": "Resolution Summary",
            "widgets": [
              {
                "textParagraph": {
                  "text": "<b>Total Duration:</b> 28 minutes<br><b>Root Cause:</b> Transient DNS routing convergence resolved; container restarted.<br><b>SLA Impact:</b> Within P1 Target SLA (≤ 4 hours)."
                }
              },
              {
                "buttonList": {
                  "buttons": [
                    {
                      "text": "Live Status Page",
                      "onClick": {
                        "openLink": { "url": "https://status.gercia.com.br" }
                      }
                    }
                  ]
                }
              }
            ]
          }
        ]
      }
    }
  ]
}

---

5. Cloudflare Health Checks & Probes Management

Synthetic probes are managed natively through the Cloudflare API or MCP tooling:

Active Synthetic Probe Registry

The platform operates 4 active synthetic edge probes querying endpoints across Southern South America (SSAM), Western North America (WNAM), and Western Europe (WEU):

Probe Identifier	Target FQDN	Architectural Scope & Component	Probe Configuration	Interval
gercia-app	app.gercia.com.br	B2B SaaS Portal for Doctors, Clinics, and Pharmacies	HTTPS GET / (200 OK)	60s
gercia-api	api.gercia.com.br	REST API Gateway, Sovereign OpenIddict STS, and AI MCP (Gercia.Server)	HTTPS GET /health (200 OK)	60s
gercia-site	gercia.com.br	Public institutional site (Cloudflare Pages)	HTTPS GET / (200 OK)	60s
gercia-docs	docs.gercia.com.br	Public Help Center and User Manuals (Cloudflare Pages)	HTTPS GET / (200 OK)	60s

Zero Trust Access Exclusions

Internal portals protected behind Cloudflare Zero Trust with mandatory Google Workspace authentication (ops.gercia.com.br) and private engineering documentation (eng.gercia.com.br) are deliberately excluded from public health check probes to prevent spurious false-positive alerts triggered by 302 OIDC redirect challenges.

CLI Probe Validation Commands

Engineers can manually trigger and inspect health check statuses:

# Query live health checks via Cloudflare API
curl -s -X GET "https://api.cloudflare.com/client/v4/zones/9b40462280303d298a2d0817c0b2b1d4/healthchecks" \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  -H "Content-Type: application/json" | jq '.result[] | {name, status, failure_reason}'

---

6. Blameless Post-Mortem Guidelines

Every P1 or P2 incident requires a formal Post-Mortem document published in docs/private/src/content/docs/observability/ within 5 business days:

1. Incident Metadata: Date, duration, severity, impacted components, and author.
2. User Impact: Percentage of impacted requests, failed prescriptions, or affected pharmacies.
3. Timeline: Chronological event log in UTC and BRT from first detection to resolution.
4. Root Cause Analysis (5 Whys): Deep systemic exploration avoiding personal blame.
5. Action Items (Preventive Engineering): Concrete work items scheduled in Linear (ENG team) to eliminate the failure mode permanently.