INTERNAL2024

SLO-Based Observability Platform

Most alerting systems page on symptoms — CPU spikes, memory pressure, error rate above threshold. The signal arrives too late and carries no context about actual service-level impact. I built an SLO-based observability layer: defined availability and latency SLOs per service, implemented multi-window burn-rate alerting (Google SRE Book Chapter 5 approach), and built Grafana dashboards showing error budget consumption in real time. The operational shift: on-call is paged only when burn rate indicates a genuine threat to SLO compliance within the measurement window — not on every transient spike. Alert noise dropped significantly, the team started treating error budgets as an engineering constraint rather than a vanity metric.

$ Architecture

→SLO definitions as code: availability (99.9% request success rate) and latency (p99 < 300ms) targets per service in version-controlled YAML, reviewed and signed off like any service contract
→Prometheus recording rules pre-computing burn rate over 1h, 6h, and 3d windows — fast-burn and slow-burn evaluation without query overhead at alert evaluation time
→Multi-window alerting: fast-burn (1h and 6h windows) triggers PagerDuty; slow-burn (3d) routes to Slack — alert noise reduced by removing single-spike false positives
→Grafana error budget dashboards: remaining budget percentage, projected exhaustion time, and per-endpoint SLO heatmap updated in real time
→ELK pipeline: Spring Boot JSON structured logs routed through Logstash to Elasticsearch; Kibana used for incident investigation with full correlation ID and trace ID join

$ Tech Stack

PrometheusGrafanaELK StackJavaSpring BootKubernetesPagerDuty

$ Outcomes

→SLO coverage across 8 production services with per-service error budget tracking and quarterly compliance reporting

→On-call alert noise reduced — multi-window burn rate eliminates single-spike false positives that previously triggered unnecessary pages

→Error budget as an engineering constraint: feature freeze and reliability work triggers defined by budget consumption, not subjective team decision

→Structured logging standard adopted org-wide: correlation IDs, trace IDs, and request context on every log line

→Automatic post-incident timeline generation from Kibana log queries and Jaeger trace spans — RCA time per incident reduced measurably