SRE Agent brings Kubernetes evidence and your runbooks together into a first AI-assisted incident diagnosis your team can review.
Non-Chrome reservation triggered shared Lumina reservation failure state.
lumina/coffeehouse-webapp-7f9bd698c-2kv8x phase=Running but container coffehouse-webapp shows ready=0.
coffehouse_webapp_runbook.md#known-issues documents this issue and provides mitigation instructions. The reservation feature does not work for Chrome based browsers, and causes the crash.
Restart the workload to clear the shared failure state: perform a rolling restart of deployment/coffeehouse-webapp in namespace sre-agent (or delete the current pod to force recreation).
After restart, confirm the container becomes Ready (ready=1) and the alert condition clears (lumina_reservation_browser_failure_active returns to 0) and Alertmanager resolves.
If the pod remains not-ready after restart, pull current pod events and container logs, and check readiness/liveness probe failures and recent config/secret changes affecting the reservation feature.
If failures recur quickly, identify the triggering client path (non-Chrome reservation flow) and temporarily block/disable that path at the edge or feature-flag level if available, while escalating to the application owner for a code-level fix.
When an alert fires, SRE Agent gathers relevant pod events, resource definitions, bounded logs, and matching runbook documentation into a unified summary for engineering review.
Prometheus or Alertmanager webhook fires with pod label context.
Read-only cluster inspection of pod events, resource limits, and recent stderr logs.
Runbook semantic retrieval & explicit uncertainty calibration based on past field reports.
Succinct diagnostic ledger card handed directly to the engineer on call.
Prometheus sends alerts through Alertmanager to SRE Agent via a standard webhook receiver. The receiver extracts Kubernetes metadata: target namespace, pod labels, container specifications, and trigger firing timestamps.
Restricted read-only RBAC queries inspect relevant pod logs, Describe events, and status transitions strictly bounded to the affected namespace and incident timeframe.
Indexes and evaluates Markdown documentation against current cluster symptoms using semantic vector retrieval and ranking against relevant documentation.
Delivers structured markdown field notes and webhook payloads summarizing confirmed symptoms, explicit uncertainties, and safe command suggestions directly to engineers for review.
Standard large language models are engineered to provide answers even when they lack definitive factual context. In infrastructure management, guessing a configuration flag or database command can cascade into critical data loss.
SRE Agent enforces hard confidence bounds. When an incident matches no pre-approved runbook in your repository, it explicitly halts inference synthesis and shifts to raw evidence compilation.
“An honest diagnostic tool displays silence and verified telemetry rather than inventing plausibility.”
Warning: No verified runbook matches error code AUTH-502; defaulting to raw evidence baseline without remediation heuristics.
SRE Agent integrates directly with verified platform telemetry and storage interfaces.
Autonomous action has no place in critical production systems. We enforce four structural barriers.
The agent operates under restrictive RBAC verbs: get, list, and watch only. Zero patch, create, or delete privileges.
Queries are hard-scoped to the pod’s specific namespace. Inspection cannot traverse across cluster boundaries or query cluster-wide secrets.
Client-side regex scrubbing redacts JWTs, Bearer tokens, DB credentials, and TLS private keys from logs before transmission to inference endpoints.
The agent generates diagnoses and exact commands, but never executes them. Remediation remains strictly under human operator control.