SRE Agent
INCIDENT DIAGNOSIS SPECIFICATION

When an alert fires, start with context.

SRE Agent brings Kubernetes evidence and your runbooks together into a first AI-assisted incident diagnosis your team can review.

DETERMINISTIC INTAKE TRACE4.1s TOTAL
  1. +00.0sINGEST
    Alertmanager Webhook Ingested
    CrashLoopBackOff · namespace: core-auth
  2. +01.4sRBAC READ
    Bounded Kubernetes Inspection
    Pod events, restart counter, bounded stderr logs
  3. +02.8sRETRIEVAL
    Runbook Vector Retrieval
    Matched team runbooks in ChromaDB / local docs
  4. +04.1sREADY
    Structured Briefing Compiled
    Factual markdown analysis synthesized for human engineer sign-off
ONGOING
INCIDENT ID: INC-2026-05-30T14:02:18Z
Incident Analysis Report

Lumina reservation browser failure

2026-05-30T14:02:18Z
fact_check Quick Summary
  • • Alert Name: LuminaReservationBrowserFailure
  • • Firing Date (UTC): 2026-05-30T14:02:18Z
  • • Severity: critical
  • • Target: lumina/coffehouse-webapp-7f9bd698c-2kv8x
description Relevant Runbooks
  • • coffehouse_webapp_runbook.md
DiagnosiS
Confidence: HIGH

The alert indicates the applicatoin has entered a shared reservation failure state triggered by a non-Chrome reservation path; the deployment is healthy/available but the single running pod's container is not ready, suggesting the app is not serving correctly despite the deployment reporting availability. Primary mitigation is to restart the coffehouse-webapp pod/deployment to clear the failure state, then verify the metric clears.

Supporting Evidence
>
[ALERT] LuminaReservationBrowserFailure is firing with description Non-Chrome reservation triggered shared Lumina reservation failure state.
>
[KUBERNETES] Deployment desired=1, ready=1, available=1; pod lumina/coffeehouse-webapp-7f9bd698c-2kv8x phase=Running but container coffehouse-webapp shows ready=0.
>
[LOGS] No container logs were captured
>
[RUNBOOKS] Runbook coffehouse_webapp_runbook.md#known-issues documents this issue and provides mitigation instructions. The reservation feature does not work for Chrome based browsers, and causes the crash.
MITIGATION STEPS
1.

Restart the workload to clear the shared failure state: perform a rolling restart of deployment/coffeehouse-webapp in namespace sre-agent (or delete the current pod to force recreation).

2.

After restart, confirm the container becomes Ready (ready=1) and the alert condition clears (lumina_reservation_browser_failure_active returns to 0) and Alertmanager resolves.

3.

If the pod remains not-ready after restart, pull current pod events and container logs, and check readiness/liveness probe failures and recent config/secret changes affecting the reservation feature.

4.

If failures recur quickly, identify the triggering client path (non-Chrome reservation flow) and temporarily block/disable that path at the edge or feature-flag level if available, while escalating to the application owner for a code-level fix.

OPERATIONAL CONTEXT

Context for the first diagnosis

When an alert fires, SRE Agent gathers relevant pod events, resource definitions, bounded logs, and matching runbook documentation into a unified summary for engineering review.

01

Firing Signal

Prometheus or Alertmanager webhook fires with pod label context.

alertname: CrashLoop
02

Gathering Boundary

Read-only cluster inspection of pod events, resource limits, and recent stderr logs.

k8s API (GET only)
03

Context Synthesis

Runbook semantic retrieval & explicit uncertainty calibration based on past field reports.

runbook vector match
04

Operator Handoff

Succinct diagnostic ledger card handed directly to the engineer on call.

Human-in-the-loop
TRIGGER: Webhook
RESULT: Structured Markdown Field Memo
OPERATIONAL PIPELINE

Four stages designed to gather evidence for review.

STAGE 01 INGESTION

Alertmanager Hook

Prometheus sends alerts through Alertmanager to SRE Agent via a standard webhook receiver. The receiver extracts Kubernetes metadata: target namespace, pod labels, container specifications, and trigger firing timestamps.

POST /api/v1/alerts/kubernetes-receiver
Header: X-Prometheus-Alertmanager-Signature
STAGE 02ISOLATION

Bounded Inspection

Restricted read-only RBAC queries inspect relevant pod logs, Describe events, and status transitions strictly bounded to the affected namespace and incident timeframe.

GET /api/v1/namespaces/{ns}/pods/{pod}/log
Scope: read_only inspection
STAGE 03EVALUATION

Runbook Evaluation

Indexes and evaluates Markdown documentation against current cluster symptoms using semantic vector retrieval and ranking against relevant documentation.

Vector Retrieval: ChromaDB Cloud
Ranking: Semantic matching & confidence evaluation
STAGE 04PRESENTATION

Human-Ready Briefing

Delivers structured markdown field notes and webhook payloads summarizing confirmed symptoms, explicit uncertainties, and safe command suggestions directly to engineers for review.

Format: Clean Markdown & Webhook JSON Payload
Operator Sign-off: Required
EVALUATION OVER HALLUCINATION

Why knowing when NO runbook matches is critical.

Standard large language models are engineered to provide answers even when they lack definitive factual context. In infrastructure management, guessing a configuration flag or database command can cascade into critical data loss.

SRE Agent enforces hard confidence bounds. When an incident matches no pre-approved runbook in your repository, it explicitly halts inference synthesis and shifts to raw evidence compilation.

Archival Design Principle

“An honest diagnostic tool displays silence and verified telemetry rather than inventing plausibility.”

warning
WARNING: NO-MATCHING-RUNBOOK

Warning: No verified runbook matches error code AUTH-502; defaulting to raw evidence baseline without remediation heuristics.

folder_open runbooks/ VECTOR SCAN: 42 DOCUMENTS
/storage/ pvc-exhaustion.md
Score: 0.21 (Below threshold)
/networking/ ingress-tls-expiry.md
Score: 0.14 (Below threshold)
×No matching candidate identified
FALLBACK ACTIVE
CONNECTED SYSTEMS

Precise Typographic Integrations Directory

SRE Agent integrates directly with verified platform telemetry and storage interfaces.

Platform / Driver
Operational Scope
Status
Kubernetes
Core API v1, Event stream correlation, Pod log extraction, CRD status interrogation.
Available Now
Prometheus & Alertmanager
Prometheus sends alerts through Alertmanager via standard webhook receiver.
Available Now
OpenAI
OpenAI API compatibility with structured JSON output.
Available Now
ChromaDB Cloud
Stores runbooks and supports semantic search across incident documentation.
Available Now
SQLite Cloud
Stores alert-derived issue history and diagnostic records.
Available Now
Datadog
Explored APM span fetching and monitor event webhook binding.
Future Exploration (Unscheduled)
AWS CloudWatch
Cross-account CloudWatch Logs Insights bounded query adapter.
Future Exploration (Unscheduled)
Jira Software
Automated ticket enrichment and post-incident action item linking.
Future Exploration (Unscheduled)
SAFETY GUARANTEES

Trust & Security Architecture

Autonomous action has no place in critical production systems. We enforce four structural barriers.

lock

Read-Only by Design

The agent operates under restrictive RBAC verbs: get, list, and watch only. Zero patch, create, or delete privileges.

crop_free

Strict Namespace Bounds

Queries are hard-scoped to the pod’s specific namespace. Inspection cannot traverse across cluster boundaries or query cluster-wide secrets.

shield

Secret Sanitization

Client-side regex scrubbing redacts JWTs, Bearer tokens, DB credentials, and TLS private keys from logs before transmission to inference endpoints.

person_check

Human in the Loop

The agent generates diagnoses and exact commands, but never executes them. Remediation remains strictly under human operator control.