Module 7 — Monitoring, Alerting & SRE with Prometheus + Grafana
Goal: deploy a full observability stack on monitor-box (Prometheus, Grafana, node_exporter, blackbox_exporter, Alertmanager), build serious dashboards, and use SLI/SLO thinking — not vibes — to decide what to alert on.
The student installs everything themselves — that's the lab. Telecoms framing throughout: this module turns lab-box into a monitored network element and the student into the NOC.
7.1 SRE concepts first (they drive everything below)
- Monitoring answers "is it broken?"; observability answers "why?". Three pillars: metrics (this module), logs (journalctl → Loki as a stretch), traces (named only).
- SLI — a measured indicator: e.g. "% of requests answered < 300 ms with non-5xx over 28 days".
- SLO — your objective for that SLI: "99.5%". Internal, engineering-owned.
- SLA — the contractual promise with penalties. Always looser than the SLO. (The student has lived on the receiving end of leased-line SLAs — use that.)
- Error budget = 100% − SLO. At 99.5%/28 days ≈ 3h 21m of allowed badness. Budget healthy → ship fast. Budget burnt → freeze features, fix reliability. This is the mechanism that ends the dev-vs-ops war.
- Alert philosophy: page only on user-visible symptoms and budget burn; everything else is a dashboard or ticket. Every page must be actionable. Alert fatigue is the NOC's "alarm storm" problem — same disease, same cure.
- The four golden signals: latency, traffic, errors, saturation.
7.2 Deploy the stack (Lab — the big one)
On monitor-box (ports 3000/9090/9093 are opened to the admin IP by the Terraform change made in module 4's checkpoint — dogfooding).
~/observability/compose.yaml:
services:
prometheus:
image: docker.io/prom/prometheus:latest
ports: ["9090:9090"]
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./alerts.yml:/etc/prometheus/alerts.yml:ro
- prom-data:/prometheus
grafana:
image: docker.io/grafana/grafana:latest
ports: ["3000:3000"]
volumes: [graf-data:/var/lib/grafana]
environment: { GF_SECURITY_ADMIN_PASSWORD: labpass }
alertmanager:
image: docker.io/prom/alertmanager:latest
ports: ["9093:9093"]
volumes: ["./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro"]
blackbox:
image: docker.io/prom/blackbox-exporter:latest
ports: ["9115:9115"]
volumes: { prom-data: {}, graf-data: {} }
prometheus.yml:
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files: [alerts.yml]
alerting:
alertmanagers:
- static_configs: [{ targets: ["alertmanager:9093"] }]
scrape_configs:
- job_name: prometheus
static_configs: [{ targets: ["localhost:9090"] }]
- job_name: node # host metrics from BOTH boxes
static_configs:
- targets: ["<lab-box-private-ip>:9100", "<monitor-box-private-ip>:9100"]
- job_name: fieldapp # the module-6 app (add /metrics in lab 7.4)
static_configs: [{ targets: ["<lab-box-private-ip>:8000"] }]
- job_name: blackbox-http # outside-in probes = closest to user experience
metrics_path: /probe
params: { module: [http_2xx] }
static_configs:
- targets:
- http://<lab-box-private-ip>:8000
- https://ratib.elhindi.net # instructor's real production site
relabel_configs:
- { source_labels: [__address__], target_label: __param_target }
- { source_labels: [__param_target], target_label: instance }
- { target_label: __address__, replacement: "blackbox:9115" }
Install node_exporter on both boxes (on each):
podman run -d --name node_exporter --net host --pid host \
-v /:/host:ro,rslave docker.io/prom/node_exporter:latest --path.rootfs=/host
Bring it up and verify targets:
cd ~/observability && podman-compose up -d
# Browser: http://<monitor-box-ip>:9090/targets → everything UP?
Debugging the inevitable DOWN targets is the real lab — security group between the boxes (fix in Terraform!), wrong private IP, exporter not running. The student uses module 1–4 skills to fix module 7 problems. Perfect.
7.3 PromQL — enough to be dangerous
In the Prometheus UI, build up gradually:
node_load1
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes # ratio
100 * (1 - avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))) # CPU%
rate(node_network_receive_bytes_total[5m]) * 8 # bps in
node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}
probe_success # blackbox up/down
probe_http_duration_seconds # per-phase latency
probe_ssl_earliest_cert_expiry - time() # seconds to cert expiry!
The one concept to hammer: counters + rate(). Counters only go up; rate()
turns them into per-second speeds. Nearly every useful query is
rate(counter[window]) plus arithmetic and sum by(...).
Generate load on lab-box and watch it appear (15s later — discuss scrape interval):
sudo apt install -y stress-ng && stress-ng --cpu 2 --timeout 120
7.4 Instrument the app — your own SLIs
Add metrics to fieldapp (swap module 5's server.py body, or do it as a PR through
the module-6 pipeline — recommended!):
# pip install prometheus-client (add to the Containerfile)
from prometheus_client import Counter, Histogram, start_http_server
REQS = Counter("app_requests_total", "Requests", ["code"])
LAT = Histogram("app_request_seconds", "Latency")
…record around each request; expose /metrics. Deploy via the pipeline, confirm
Prometheus scrapes it, then compute real service SLIs:
# Availability SLI: share of non-5xx
sum(rate(app_requests_total{code!~"5.."}[5m])) / sum(rate(app_requests_total[5m]))
# Latency SLI: p95 from the histogram
histogram_quantile(0.95, rate(app_request_seconds_bucket[5m]))
Drive traffic with a loop or hey/ab, then inject failures (add a route that
returns 500 for 10% of requests — one line) and watch the SLI drop.
7.5 Grafana — extensive dashboarding
Add Prometheus as a data source (http://prometheus:9090), then:
- Import community dashboards first — dashboard ID 1860 (Node Exporter Full) and 7587 (blackbox). Explore what "good" looks like; the student maps panels back to PromQL by clicking Edit on each.
- Build the "Field App — Service Health" dashboard from scratch (this is the assessed artifact):
- Row 1 — golden signals: traffic (req/s), error rate %, p50/p95/p99 latency, saturation (CPU% + memory of lab-box).
- Row 2 — SLO: availability SLI gauge with SLO threshold line (99.5%),
error-budget-remaining stat panel:
1 - ((1 - sli_28d) / (1 - 0.995))style calculation, and burn-down over time. - Row 3 — dependencies: blackbox probe success + duration for the app and the instructor's production site; TLS cert days-remaining stat.
- Variables:
$instancetemplating so one dashboard serves both boxes. - Thresholds/colours: green/amber/red bound to SLO maths, not gut feel.
- Export the dashboard JSON and commit it to the repo — dashboards are code too.
Reference dashboard: a completed version of the assessed dashboard ships in this repo at
grafana/fieldapp-service-health.json(also downloadable from the course site at/grafana/fieldapp-service-health.json). Import it via Dashboards → New → Import → Upload JSON, and select your Prometheus data source when prompted. Two legitimate ways to use it: build yours from scratch and compare afterwards (preferred), or import it first and earn the pass by explaining every panel's PromQL and threshold to the instructor. It includes the$instancevariable, an adjustable$slo_window(1h–28d — use short windows on day one, since 28d needs 28 days of data), the error-budget maths, burn-rate panel, blackbox probe status and the TLS expiry stat.
7.6 Alerting — page on symptoms, tied to the SLO
alerts.yml:
groups:
- name: service
rules:
- alert: AppDown
expr: probe_success{instance=~".*8000.*"} == 0
for: 2m
labels: { severity: page }
annotations: { summary: "Field app is failing outside-in probes" }
- alert: HighErrorRate
expr: |
sum(rate(app_requests_total{code=~"5.."}[5m]))
/ sum(rate(app_requests_total[5m])) > 0.02
for: 5m
labels: { severity: page }
annotations: { summary: "Error rate above 2% — burning error budget fast" }
- alert: CertExpirySoon
expr: probe_ssl_earliest_cert_expiry - time() < 14 * 86400
labels: { severity: ticket }
annotations: { summary: "TLS cert expires in under 14 days" }
- alert: DiskFillingUp
expr: predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 24*3600) < 0
for: 30m
labels: { severity: ticket }
annotations: { summary: "Root disk predicted full within 24h" }
alertmanager.yml (email or a webhook to a free Slack/Discord — student's choice):
route severity=page loudly, severity=ticket quietly. Discuss why the routing
split is the whole point.
Fire drill (assessed)
- Instructor kills fieldapp on lab-box (
podman stop fieldapp) without warning. - Student is "on call": sees the page, works the dashboard, finds the cause via
podman ps/journalctl, restores service (redeploy via pipeline!), and then writes a blameless post-incident review: timeline, impact (minutes of error budget spent), root cause, actions. Template incapstone-app/PIR-template.md. - Repeat later with subtler sabotage: the 10%-500s route enabled via env var (alert fires, app looks "up"), or disk-fill on monitor-box (the monitoring monitors itself — who watches the watchmen discussion).