The Engineer's Watch

What a process actually is

A running service is a process with an identity and a set of budgets: a user and group, a working directory, an environment, open file descriptors, memory, and a share of the CPU. Nearly every mysterious production failure is one of those budgets running out.

• File descriptors. Every socket, file and pipe is one, and the default limit is often 1,024. A busy server with connection pools and keep-alive connections passes that easily, and then fails with "too many open files", usually while accepting new connections, which looks like a network problem.
• Memory. The kernel will hand out more than it has and hope; when it runs out, the OOM killer picks a process and kills it with SIGKILL. Under a cgroup limit, your container is chosen first, and the exit status is 137, which is 128 + 9.
• CPU. A quota is enforced by throttling: the process is stopped for the rest of each 100 ms period once it has used its share, which shows up as latency, not as high CPU.
• Threads and tasks. A limit on tasks turns "can't create thread" into a crash under load.

systemd: the thing that runs your thing

On a plain Linux host, systemd is process 1. It starts your service, restarts it when it dies, applies the limits above through cgroups, collects its output into the journal, and orders it against other units. A unit file is the whole operational contract in one page, which is why it's worth writing carefully rather than copying.

[Unit]
Description=Invoicing API
After=network-online.target
Wants=network-online.target
# Flapping protection lives here, not in [Service]: give up after 5 starts in a minute.
StartLimitIntervalSec=60
StartLimitBurst=5

[Service]
Type=exec
User=invoicing
Group=invoicing
WorkingDirectory=/srv/invoicing
# Secrets in a root-owned 0600 file, not in the unit: unit files are world-readable.
EnvironmentFile=/etc/invoicing/env
ExecStart=/srv/invoicing/venv/bin/gunicorn --bind 127.0.0.1:8080 app:application
Restart=on-failure
RestartSec=2s
TimeoutStopSec=30s
LimitNOFILE=65535
MemoryHigh=1200M
MemoryMax=1500M
CPUQuota=200%
TasksMax=512
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/invoicing
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Three lines there deserve attention. MemoryHigh below MemoryMax means the kernel throttles and reclaims before it kills, which turns some out-of-memory kills into slowness you can alert on. Restart=on-failure with a start limit stops a crash loop from hammering a database for hours. And ProtectSystem=strict makes the filesystem read-only except the paths you name, so a compromised or confused process can't rewrite the host.

The commands you'll use daily are few: systemctl status invoicing-api for state and recent logs, systemctl restart, journalctl -u invoicing-api -f to follow output, systemctl cat to see the effective unit, and systemd-analyze verify to check one before it goes near a machine.

Shutdown is part of your program

Stopping a service is a negotiation. systemd sends SIGTERM and waits TimeoutStopSec; Kubernetes does the same with terminationGracePeriodSeconds, 30 seconds by default. If the process hasn't exited by then, it gets SIGKILL, which it cannot catch: in-flight requests die, buffers are lost, half-written files stay half-written.

import queue
import signal
import sys
import threading

jobs: queue.Queue = queue.Queue()
stopping = threading.Event()


def on_sigterm(signum, frame):
    stopping.set()          # the handler does one thing: raise a flag


signal.signal(signal.SIGTERM, on_sigterm)


def run_worker(process, poll_s=0.2):
    """Take no new work after SIGTERM, finish the job in hand, then exit cleanly."""
    while not stopping.is_set():
        try:
            job = jobs.get(timeout=poll_s)
        except queue.Empty:
            continue
        process(job)        # a job that started before the signal runs to completion
        jobs.task_done()
    sys.exit(0)             # unclaimed work stays on the queue for the next worker

For an HTTP server the shape is the same, with one extra step at the front: fail the readiness check first, wait a few seconds for load balancers to stop routing to you, and only then stop accepting. The Engineer's Fleet, chapter 15, has the full ordering. Two rules make the rest work: keep individual units of work shorter than the grace period, and make them idempotent, because sooner or later one will be cut in half.

Where things are, and who you are

Run as a dedicated unprivileged user, never root. A service that needs a port below 1024 doesn't need root for it: give the binary the CAP_NET_BIND_SERVICE capability, or bind to 8080 and let a proxy or a load balancer map the port. Configuration belongs in /etc, variable state in /var/lib, logs in the journal or /var/log, and runtime facts about processes in /proc, where /proc/<pid>/limits, /proc/<pid>/status and /proc/<pid>/fd answer most "what is it actually doing" questions.

In a container, write logs to standard output and let the platform collect them. On a plain host, write to the journal or to files with logrotate configured. A disk filled by logs is one of the most common self-inflicted outages there is, and it takes down everything on the machine at once, usually starting with the database.

[Unit]
Description=Payments worker

[Service]
Environment=DATABASE_URL=postgres://app:hunter2@db/payments
ExecStart=/usr/bin/python3 /opt/payments/worker.py
Restart=always
RestartSec=0
KillSignal=SIGKILL

[Install]
WantedBy=multi-user.target

The first minute

Netflix's performance team published a checklist of what to run in the first sixty seconds on an unfamiliar, unhappy Linux box. It's still the fastest way to orient yourself, and it costs nothing to learn.

Two of those numbers are routinely misread. Load average on Linux counts processes waiting for disk as well as for CPU, so a load of 40 on eight cores can mean a saturated disk and an idle processor. And steal time, the st column in vmstat, is time your virtual machine wanted the CPU and the hypervisor gave it to someone else: your code didn't get slower, your neighbour got busier.

The USE method

Brendan Gregg's USE method turns "the box is slow" into a finite checklist. For every resource, check utilisation (how much of the time it's busy), saturation (how much work is queued for it) and errors. Saturation is the one people skip, and it's usually where the answer is: a disk at 100% utilisation with no queue is fine, and a disk at 60% with thirty requests waiting is the outage.

Running, or waiting?

Before reaching for any profiler, answer one question: is the slow thing burning CPU, or waiting for something else? The two have completely different tool sets, and you can tell them apart by comparing CPU time with wall-clock time.

import hashlib
import resource
import time


def where_did_the_time_go(fn, *args):
    """Wall time against CPU time: is this work running, or waiting for something?"""
    r0, t0 = resource.getrusage(resource.RUSAGE_SELF), time.monotonic()
    fn(*args)
    r1, t1 = resource.getrusage(resource.RUSAGE_SELF), time.monotonic()
    wall = t1 - t0
    cpu = (r1.ru_utime - r0.ru_utime) + (r1.ru_stime - r0.ru_stime)
    return wall, cpu, cpu / wall

# hashing  wall 0.42 s  cpu 0.42 s  on-CPU 98%   -> profile the code
# waiting  wall 0.51 s  cpu 0.00 s  on-CPU  0%   -> find out what it is waiting for

High CPU means a CPU profile, best read as a flame graph: stacks on the x-axis sorted alphabetically, depth on the y-axis, width proportional to time. You read it by looking for wide plateaus, which are where the time goes, and ignoring the height. Near-zero CPU with high latency means the process is blocked, and the question becomes what on: a lock, a disk, a database, a DNS lookup, a downstream service. That's off-CPU analysis, done with off-CPU flame graphs, with strace -c -p <pid> for a syscall summary, or with the application's own thread dump.

Modern Linux tooling for both is eBPF: bpftrace for one-liners and short scripts, the bcc collection for ready-made tools such as biolatency, execsnoop and offcputime. These attach to a running kernel with no restarts and low overhead, which is why they've become the default way to see inside a production box.

The parts people forget

• Sockets and backlogs. ss -ltn shows listening sockets with Recv-Q, the number of connections waiting to be accepted, and Send-Q, the accept queue's size. A full accept queue silently drops connections: clients see timeouts while your server looks idle.
• Connection tracking and ports. A NAT gateway or firewall with a full connection-tracking table, or a client that has exhausted its ephemeral ports, produces failures that look like packet loss.
• DNS. Slow or failing name resolution is one of the most common causes of "everything is a bit slow", and it rarely appears on a dashboard.
• The container's own limits. Inside a cgroup, the numbers in top describe the host, not you. Read memory.current against memory.max, and cpu.stat, whose nr_throttled and throttled_usec tell you how often the kernel stopped you at the ceiling.

Working the problem

During an incident the method is narrower than this: mitigate first, diagnose second (chapter 16). But whenever you are actually diagnosing, three habits pay for themselves. Form a hypothesis before running the command, so the output means something. Change one thing at a time, or you'll never know which change worked. And write down what you ran and what it said, because in two hours you will not remember, and the postmortem will need it.

Two laws worth knowing by heart

Little's law says that the average number of requests in flight equals the arrival rate times the average time each one takes: L = λW. It holds for any stable system, whatever it does inside. At 500 requests a second with an average latency of 200 ms, 100 requests are in your system at any moment, so a connection pool of 40 is a bottleneck you built yourself. Run it backwards to size things: 64 workers, each taking 200 ms, can serve 320 requests a second and not one more.

The second law is what happens as you approach that ceiling. For a simple queue, the response time is the service time divided by the idle fraction: W = S / (1 − ρ). The shape of that curve is the single most useful thing to know about capacity.

Percentiles, and what they hide

An average latency is the least interesting number you can compute, because it describes nobody: it's dragged around by both the fast majority and the slow tail. Percentiles describe users. The p99 is the experience of the unluckiest request in a hundred, and on a page that makes twenty requests, most users meet it several times a session (The Engineer's Fleet, chapter 14).

Two rules follow. Percentiles cannot be averaged. The mean of ten instances' p99s is not the fleet's p99, and the error is not small. In a run with nine healthy instances and one sick one, the mean of the ten p99s was 152 ms while the true p99 across all the requests was 916 ms: averaging hid a sick instance almost perfectly. To get a real percentile you need the underlying distribution, which is why metrics systems store histograms (chapter 11). And percentiles are not additive: a request that fans out to five services, each with a 1% slow tail, is slow far more often than 1% of the time.

Your load generator is probably lying

Gil Tene named the most common measurement bug coordinated omission. A load generator with a fixed number of workers sends the next request only when the last one returns. When the server stalls, the generator stops sending, so the requests that would have hit the stall are never made, never measured, and never appear in the tail. The result is a beautiful latency report from a system that was unusable.

def closed_loop(port, seconds):
    lat, end = [], time.monotonic() + seconds
    while time.monotonic() < end:
        t0 = time.monotonic(); one_request(port); lat.append(time.monotonic() - t0)
    return lat


def open_loop(port, seconds, rate):
    lat, lock, threads = [], threading.Lock(), []
    start = time.monotonic()
    def fire(intended):
        try:
            one_request(port)
        except Exception:
            return
        with lock:
            lat.append(time.monotonic() - intended)     # measured from when it SHOULD have been sent
    for i in range(int(seconds * rate)):
        intended = start + i / rate
        delay = intended - time.monotonic()
        if delay > 0:
            time.sleep(delay)
        t = threading.Thread(target=fire, args=(intended,), daemon=True)
        t.start(); threads.append(t)
    for t in threads:
        t.join(timeout=10)
    return lat

Pointed at the same server, a toy that serves in 5 ms but stalls for 200 ms once a second, the two generators describe different worlds:

Same server, same stalls. One report says one request in a hundred was slow; the other says more than a third were. The open-loop generator is right, because real users arrive whether or not you're ready for them, and it measures each request from the moment it should have been sent. Use generators that hold a fixed arrival rate and correct for this, such as k6, Gatling, wrk2 or Vegeta, and be suspicious of any tool whose concurrency you set instead of its rate.

Load testing that tells you something

• Test at rates, not at concurrency, and ramp: 10%, 25%, 50%, 75%, 100%, 125% of expected peak. You're looking for the knee in the curve above, not a single pass or fail.
• Find the breaking point deliberately, so you know what happens beyond it: graceful degradation, or collapse.
• Use realistic data and cache states. A test against a warm cache and a thousand-row table measures nothing.
• Watch the generator too. If it's CPU-bound or running out of ports, you're measuring the generator.
• Record the system's own metrics during the run, not just the client's: utilisation, saturation and queue depths tell you which resource broke first.

Capacity planning, on one page

Capacity is arithmetic once you have a number for what one instance can do. Suppose peak traffic is 12,000 requests a second, and load testing puts the knee for one instance at 800 a second. Run instances at 60% of that, keeping the latency multiplier near two:

Then check the same arithmetic against the things that don't autoscale: database connections, the connection limit on the database itself, queue consumers, third-party rate limits, and licences. The bottleneck at 44 instances is rarely the instances.

What a container actually is

Three kernel features, combined. Namespaces give the process its own view of process IDs, network interfaces, mounts, hostname and users. Cgroups give it the budgets from chapter 1: memory, CPU, tasks, I/O. And an image provides the root filesystem it sees. The kernel is the host's, shared with every other container on the machine, which is why a container is a weaker isolation boundary than a virtual machine, and why a kernel-level limit or bug affects everything on the box at once.

Two consequences bite immediately. Your process usually runs as PID 1, which on Linux has no default signal handlers: if your program doesn't handle SIGTERM itself, it simply ignores it and gets killed at the end of the grace period, exactly as in chapter 1. PID 1 is also expected to reap orphaned child processes, which most application runtimes don't do, so a container that spawns subprocesses accumulates zombies unless you add a tiny init such as tini, or run the container with --init.

Layers, and why build order matters

Each instruction in a Dockerfile produces a layer, and layers are cached: a build reuses every layer up to the first one whose inputs changed, then rebuilds the rest. That single rule dictates the order of a good Dockerfile: things that rarely change first, things that change every commit last.

A production Dockerfile

A multi-stage build compiles or installs in one stage and copies only the result into a minimal final image, so the toolchain, caches and build secrets never ship. The final stage here is a distroless image: an operating system layer with the language runtime and nothing else, no shell and no package manager, which removes most of what a scanner would otherwise find and most of what an attacker would otherwise use.

# syntax=docker/dockerfile:1
FROM python:3.13-slim@sha256:1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a AS build
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --target /deps -r requirements.txt

FROM gcr.io/distroless/python3-debian12@sha256:2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b
WORKDIR /app
ENV PYTHONPATH=/deps PYTHONUNBUFFERED=1
COPY --from=build /deps /deps
COPY src/ ./src/
USER 10001
EXPOSE 8080
ENTRYPOINT ["python", "-m", "src.server"]

Points worth copying: the base images are pinned by digest, because tags move and :latest means "whatever was pushed last"; the container runs as a numeric non-root user, which works even where the image has no /etc/passwd; the entrypoint is in exec form, so the process is PID 1 and receives signals rather than being wrapped in a shell that swallows them; and a .dockerignore keeps .git, local virtual environments and test fixtures out of the build context.

Build-time attestations are worth turning on: docker buildx build --sbom=true --provenance=true attaches a software bill of materials and a record of how the image was built, which is what lets you answer "is this image affected?" during the next dependency emergency without guessing. Scan images in the pipeline, and rebuild regularly even when your code hasn't changed, because the base image's vulnerabilities accumulate whether or not you deploy.

Make the rules a check

Habits that depend on everyone remembering them don't survive a busy week. Most of the advice above can be enforced in a few dozen lines, run in CI on every pull request.

"""Fail the build on Dockerfile habits that hurt in production."""
import re
import sys

SECRETISH = re.compile(r"(SECRET|TOKEN|PASSWORD|PASSWD|API_?KEY)\s*=", re.I)


def instructions(text):
    """(line number, instruction, argument), with continuations joined."""
    out, buf, start = [], "", 0
    for n, raw in enumerate(text.splitlines(), 1):
        line = raw.split("#")[0].rstrip() if not raw.lstrip().startswith("#") else ""
        if not line.strip():
            continue
        if not buf:
            start = n
        buf += line.rstrip("\\")
        if line.endswith("\\"):
            continue
        verb, _, arg = buf.strip().partition(" ")
        out.append((start, verb.upper(), arg.strip()))
        buf = ""
    return out


def check(text):
    ins = instructions(text)
    froms = [(n, a) for n, v, a in ins if v == "FROM"]
    last_from = froms[-1][0] if froms else 0
    problems = []
    for n, image in froms:
        if "@sha256:" not in image:
            problems.append(f"{n}: pin the base image by digest, not by tag: {image.split()[0]}")
    users = [a for n, v, a in ins if v == "USER" and n > last_from]
    if not users or users[-1] in ("root", "0"):
        problems.append(f"{last_from}: the final stage runs as root; add a USER")
    for n, v, a in ins:
        if v in ("ENTRYPOINT", "CMD") and not a.startswith("["):
            problems.append(f"{n}: use the exec form of {v} so the process receives SIGTERM")
        if v == "ADD" and not a.startswith(("http://", "https://")):
            problems.append(f"{n}: use COPY unless you need ADD's URL or archive handling")
        if v in ("ENV", "ARG") and SECRETISH.search(a):
            problems.append(f"{n}: secrets do not belong in an image layer")
        if v == "RUN" and "apt-get install" in a and "rm -rf /var/lib/apt/lists" not in a:
            problems.append(f"{n}: clean the package lists in the same RUN, or they stay in the layer")
    copies = [n for n, v, a in ins if v == "COPY" and a.split()[0] in (".", "./")]
    installs = [n for n, v, a in ins if v == "RUN" and re.search(r"\b(pip|npm|poetry|uv|go mod|bundle)\b", a)]
    if copies and installs and min(copies) < max(installs):
        problems.append(f"{min(copies)}: copy dependency manifests before the source, or every "
                        "code change rebuilds the dependency layer")
    return problems


if __name__ == "__main__":
    found = [p for path in sys.argv[1:] for p in check(open(path).read())]
    print("\n".join(found) or "Dockerfile looks fine")
    sys.exit(1 if found else 0)

Running them safely

• Read-only root filesystem, with a writable volume for the few paths that need it.
• Drop all capabilities and never run privileged. If you need to bind a low port, map it outside instead.
• Set memory and CPU budgets for every container, for the reasons in chapters 1 and 2.
• Rootless where you can. Rootless runtimes map the container's root to an unprivileged host user, so an escape lands nowhere useful.
• Don't patch running containers. An image you edited in production no longer matches anything you scanned, signed or can rebuild.

FROM python:latest
ENV DATABASE_PASSWORD=hunter2
ADD . /app
WORKDIR /app
RUN apt-get update && apt-get install -y build-essential
RUN pip install -r requirements.txt
CMD python -m src.server

1: pin the base image by digest, not by tag: python:latest
1: the final stage runs as root; add a USER
2: secrets do not belong in an image layer
3: use COPY unless you need ADD's URL or archive handling
5: clean the package lists in the same RUN, or they stay in the layer
7: use the exec form of CMD so the process receives SIGTERM

The model

Every object you create is a record in the API server, backed by etcd. Controllers watch those records and work to make reality match: the Deployment controller creates a ReplicaSet per revision, the ReplicaSet controller creates pods, the scheduler assigns each pod to a node, and the kubelet on that node starts containers and reports back. Nothing is a command; everything is a statement of intent that something else reconciles, continuously. That is why deleting a pod doesn't reduce your capacity for long, and why a bad declaration recreates itself no matter how many times you delete the symptom.

Objects find each other by labels, never by name. A Service sends traffic to whichever pods carry the labels in its selector and pass their readiness probe.

A Deployment worth copying

apiVersion: apps/v1
kind: Deployment
metadata:
  name: invoicing-api
  labels: { app: invoicing-api }
spec:
  replicas: 6
  revisionHistoryLimit: 5
  progressDeadlineSeconds: 600          # fail the rollout rather than hanging forever
  strategy:
    type: RollingUpdate
    rollingUpdate: { maxSurge: 2, maxUnavailable: 0 }   # never go below the current capacity
  selector:
    matchLabels: { app: invoicing-api }
  template:
    metadata:
      labels: { app: invoicing-api }
    spec:
      terminationGracePeriodSeconds: 45  # must exceed your longest request, plus the preStop wait
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels: { app: invoicing-api }
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        seccompProfile: { type: RuntimeDefault }
      containers:
        - name: api
          image: registry.example.com/invoicing/api@sha256:3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c
          ports:
            - { name: http, containerPort: 8080 }
          envFrom:
            - configMapRef: { name: invoicing-api-config }
            - secretRef: { name: invoicing-api-secrets }
          resources:
            requests: { cpu: 250m, memory: 512Mi }   # the reservation the scheduler honours
            limits: { memory: 512Mi }                # no CPU limit: requests already protect neighbours
          startupProbe:                              # slow starts don't count as failures
            httpGet: { path: /healthz, port: http }
            periodSeconds: 2
            failureThreshold: 30
          readinessProbe:                            # may check dependencies: traffic stops, pod lives
            httpGet: { path: /readyz, port: http }
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:                             # must NOT check dependencies: failure means restart
            httpGet: { path: /healthz, port: http }
            periodSeconds: 10
            failureThreshold: 6
          lifecycle:
            preStop:
              sleep: { seconds: 5 }                  # let endpoints propagate before SIGTERM
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }
          volumeMounts:
            - { name: tmp, mountPath: /tmp }
      volumes:
        - name: tmp
          emptyDir: {}
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: invoicing-api
spec:
  maxUnavailable: 1                      # node drains take one pod at a time, not all six
  selector:
    matchLabels: { app: invoicing-api }

Requests, limits and what the kernel does with them

A request is a reservation: the scheduler only places a pod on a node with that much unclaimed CPU and memory, and it decides which pods get evicted first when a node runs short. A limit is a ceiling enforced by the kernel, and the two resources behave completely differently at that ceiling. Exceeding a memory limit gets the container killed (chapter 1); reaching a CPU limit gets it throttled inside every 100 ms period, which shows up purely as latency (chapter 2).

That asymmetry drives the usual advice: set memory requests equal to memory limits, so the pod is predictable and gets the highest quality-of-service class, and be very careful with CPU limits, which throttle even when the node is idle. Set CPU requests properly instead, and if you do set a limit, leave real headroom above the request. Since version 1.35, Kubernetes can resize a running pod's CPU and memory in place, without restarting it, which finally makes vertical autoscaling usable for long-lived workloads.

Probes: three questions, not one

• Startup: "has it finished booting?" Until it passes, the other probes are suspended, so a slow start isn't mistaken for a failure.
• Readiness: "should it get traffic now?" Failing takes the pod out of the Service and leaves it alone. It's allowed to consider dependencies.
• Liveness: "is it broken beyond recovery?" Failing restarts the container, so it must check only the process itself, never a dependency. A liveness probe that checks the database restarts every pod in the fleet the moment the database is slow, which is how a database hiccup becomes an outage.

Around shutdown, one race matters. When a pod is deleted, the kubelet sends SIGTERM and the endpoint controller removes it from the EndpointSlice at the same time, and those propagate independently. A short preStop sleep, as in the manifest, gives routing time to catch up so no request is sent to a process that has already begun shutting down.

Keeping the fleet spread and available

A PodDisruptionBudget constrains voluntary disruptions, such as node drains during a cluster upgrade, so the platform can't take all your pods at once. Topology spread constraints keep replicas across zones and nodes, so a lost zone doesn't take a whole service (The Engineer's Fleet, chapter 15). Neither is on by default, and both matter most on the day someone upgrades the cluster.

For scaling, three controllers do different jobs: the HorizontalPodAutoscaler adds pods based on a metric, KEDA does the same from queue depth or external events, and a node autoscaler such as Karpenter or the cluster autoscaler adds machines when pods won't fit. A common mistake is to scale pods on CPU when the real signal is queue depth or concurrency.

Getting traffic in

Inside the cluster, a Service gives a stable virtual address and DNS name. From outside, you need an ingress controller or a cloud load balancer. This is one area where 2026 forced everyone's hand: ingress-nginx, by far the most widely deployed ingress controller, was retired by the Kubernetes project and reached end of life on 24 March 2026, after repeated critical vulnerabilities and a maintainer shortage. The Ingress API itself remains, and other controllers are unaffected, but new work belongs on the Gateway API, which replaces a pile of annotations with typed resources for routes, and has a supported migration tool. If you still run ingress-nginx, migrating is not a 2027 project.

When something is wrong

The order that answers most questions: kubectl describe pod, whose Events section explains scheduling failures, image pull errors, probe failures and OOM kills; kubectl logs --previous for the container that just died; kubectl get events --sort-by=.lastTimestamp for the cluster's own story; and kubectl debug to attach an ephemeral container with a shell to a distroless pod that has none. CrashLoopBackOff is not an error, it's the platform backing off between restarts; the error is in the previous container's logs.

The largest outages are configuration changes

On 18 November 2025 a permissions change in one of Cloudflare's databases caused a generated file used by its bot-management system to double in size. The file was loaded by the proxy that fronts the network, which had a fixed limit on how large it could be, and the proxy began returning 5xx for a large part of the internet. It took hours to resolve. No code was deployed; a piece of generated data changed.

That shape recurs everywhere: a value nobody reviewed, applied everywhere at once, hitting a limit nobody remembered. Configuration deserves the same care as a deploy, which means review, staged rollout, limits with sane behaviour when exceeded, and a way back.

Three kinds of configuration

• Baked in at build time: compiled defaults and the code itself. Changing it means a new artefact, so it's the slowest and safest.
• Read at start-up: environment variables and mounted files. Changing it means a restart, so it's covered by your normal rollout (chapter 10).
• Changed at run time: feature flags, dynamic config, rule and model files. This is the fastest, the most useful, and the most dangerous, because it bypasses every safeguard the deployment pipeline gives you unless you rebuild those safeguards around it.

Put settings in the slowest category that still meets the need, and give anything in the third category a staged rollout, a verification step and an obvious rollback.

Validate at start-up, and say everything that's wrong

The usual pattern, int(os.environ.get("WORKERS", "4")) scattered through the code, fails in the worst possible way: a typo becomes a default, a bad value fails at the first request rather than at boot, and nobody can list what the service actually reads. Parse the whole configuration once, at start-up, and refuse to run if it's wrong.

import os
from dataclasses import MISSING, dataclass, fields


class ConfigError(Exception):
    """Raised once, listing everything that is wrong."""


def _parse(kind, raw):
    if kind is bool:
        if raw.lower() not in ("true", "false", "1", "0"):
            raise ValueError("expected true or false")
        return raw.lower() in ("true", "1")
    return kind(raw)


@dataclass(frozen=True)
class Config:
    database_url: str                        # no default: it must be supplied
    port: int = 8080
    workers: int = 4
    request_timeout_s: float = 5.0
    new_pdf_renderer: bool = False

    def redacted(self) -> dict:
        out = {}
        for f in fields(self):
            value = getattr(self, f.name)
            out[f.name] = "***" if any(w in f.name for w in ("url", "password", "token", "key")) else value
        return out


CHECKS = {
    "port": (lambda v: 1 <= v <= 65535, "must be between 1 and 65535"),
    "workers": (lambda v: v >= 1, "must be at least 1"),
    "request_timeout_s": (lambda v: v > 0, "must be positive"),
}


def load(env=None) -> Config:
    """Read the whole configuration, then report every problem at once and stop."""
    env = os.environ if env is None else env
    values, problems = {}, []
    for f in fields(Config):
        raw = env.get(f.name.upper())
        if raw is None or raw == "":
            if f.default is MISSING:
                problems.append(f"{f.name.upper()} is required")
            continue
        try:
            value = _parse(f.type, raw)
        except ValueError as exc:
            problems.append(f"{f.name.upper()}={raw!r}: {exc}")
            continue
        ok, why = CHECKS.get(f.name, (lambda v: True, ""))
        if not ok(value):
            problems.append(f"{f.name.upper()}={value}: {why}")
            continue
        values[f.name] = value
    if problems:
        raise ConfigError("bad configuration:\n  " + "\n  ".join(problems))
    return Config(**values)

A bad environment now produces one clear failure before the process serves anything:

ConfigError: bad configuration:
  DATABASE_URL is required
  PORT=70000: must be between 1 and 65535
  WORKERS=0: must be at least 1
  REQUEST_TIMEOUT_S=-1.0: must be positive
  NEW_PDF_RENDERER='yes': expected true or false

Note that it reports every problem, not the first: fixing configuration one error per deploy is a miserable way to spend an afternoon. Log the effective configuration at start-up with secrets redacted, as redacted() does, so an incident never turns into an argument about which value was in force.

One artefact, many environments

The same image runs in staging and production, with different configuration supplied to it; environments differ in values, never in build. Keep the values in version control next to the code that reads them, apply them through the pipeline rather than by hand, and keep the difference between environments small enough to print. A staging environment configured differently from production tests something other than production.

Secrets

Everything above applies to secrets, plus a set of rules of its own. Stage 13 covers the cryptography and the threat model; these are the operational parts.

• Never in the image, the repository, or a unit file. Images are shared, repositories are cloned, and layers keep deleted files (chapter 4).
• Prefer short-lived credentials over stored ones. Workload identity lets a running service exchange its platform identity for a token that expires in minutes, and lets a pipeline authenticate to a cloud without a long-lived key in a CI variable. A secret that doesn't exist can't leak.
• Where you must store one, use a secrets manager with access control, audit logs and rotation, and pull it at start-up or mount it as a file. Files are usually better than environment variables, because environments are inherited by child processes and dumped by crash reporters.
• Kubernetes Secrets are base64, not encryption. Enable encryption at rest, restrict access with RBAC, and consider an operator that syncs from a real secrets manager.
• Rotate on a schedule, not after an incident, and make sure rotation is a routine that works: a credential nobody can rotate is a credential you can never revoke.

import os

DEBUG = os.environ.get("DEBUG", "false") == "true"
TIMEOUT = int(os.environ.get("TIMEOUT", 5))
DATABASE_URL = os.environ.get("DATABSE_URL", "postgres://localhost/dev")
STRIPE_KEY = os.environ.get("STRIPE_KEY", "sk_test_abc123")
print(f"starting with {os.environ}")

Compute you can lose

The first mental shift is that instances are disposable. They get rebooted for host maintenance, retired with two minutes' notice, or reclaimed in seconds if they're spot capacity. Anything that must survive belongs in storage, not on a disk you attached by hand, and anything that takes an hour to rebuild by hand is an outage waiting for its moment. Build machines from images and code (chapter 8), spread them across zones, and let the platform replace them.

Spot or preemptible instances are typically a fraction of the on-demand price and suit anything that can be interrupted: batch jobs, queue workers, CI runners, stateless capacity above a baseline. Mixing a committed baseline with spot for the peak is where most of the compute savings in chapter 19 come from.

Storage you can't

• Block storage is a virtual disk with a size, an IOPS allowance and a throughput allowance, sometimes with burst credits that run out. Very many "the database got slow after three weeks" mysteries are exhausted burst credits (chapter 2).
• Object storage is the default home for anything large or numerous: uploads, generated PDFs, backups, logs. It's cheap, effectively unlimited, and durable across zones, with lifecycle rules that move old objects to colder, cheaper tiers.
• Managed databases give you backups, patching, replicas and failover. You still own the schema, the queries, the connection limits, major-version upgrades, and the knowledge that a failover takes tens of seconds during which writes fail.

A network you have to design

Within that shape, the parts to get right are: security groups as the real firewall, written as "this service may talk to that service" rather than to address ranges; private endpoints so traffic to managed services doesn't leave the network; and an awareness that cross-zone traffic is charged, which quietly makes chatty service-to-service calls a line item as well as a latency problem.

Identity is the control plane

Everything in a cloud is an API call, and IAM decides which calls succeed. Two rules carry most of the weight. Grant the narrowest permission that works, scoped to specific resources and prefixes. And stop storing long-lived credentials: give workloads an identity the platform issues and rotates, and let pipelines exchange a short-lived OIDC token for a role rather than holding an access key in a CI variable.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadWriteThisTenantsPdfs",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::invoicing-pdfs-prod/tenants/*"
    },
    {
      "Sid": "ListOnlyUnderThatPrefix",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::invoicing-pdfs-prod",
      "Condition": { "StringLike": { "s3:prefix": "tenants/*" } }
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com" },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
          "token.actions.githubusercontent.com:sub": "repo:acme/invoicing:ref:refs/heads/main"
        }
      }
    }
  ]
}

The condition in the second policy is the whole security of it. Without the sub condition, any repository on that provider could assume the role; with a wildcard such as repo:acme/*, any repository in your organisation could, including one a contractor forked. Exact matches, then.

Quotas, regions and the limits you can't see

Every cloud enforces quotas you didn't choose: instances per region, IP addresses, API calls per second, open connections on a load balancer. They're invisible until a scale-up stops at a round number. Find the ones that apply to your growth, ask for increases before you need them, and alert on usage against quota the way you alert on disk.

On geography: a zone is a failure domain within a region, and running in three of them is the standard defence (The Engineer's Fleet, chapter 15). A region is a much bigger step, involving data residency, replication and latency, and multi-region is a project, not a checkbox. Multi-cloud is another order of magnitude again, and usually buys less resilience than spending the same effort on backups you have actually restored (chapter 18).

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": "s3:*", "Resource": "*" },
    { "Effect": "Allow", "Action": ["iam:PassRole", "kms:Decrypt"], "Resource": "*" }
  ]
}

What you get for the trouble

Writing infrastructure as code costs time up front and repays it in four ways. You can rebuild: a region, an environment or a deleted resource comes back from a repository rather than from memory. You can review: a change to a security group is a pull request with a diff, an author and a reason. You can audit: what production looks like is knowable without logging in. And your environments can actually match, which is what makes testing in staging mean anything.

The tools are declarative. You describe the state you want; the tool compares it with a record of what it built, computes a plan, and applies it. That record, the state file, is the only thing that ties your code to real resources, which makes it the most important file in the repository and the one you never edit by hand. Keep it in a remote backend with locking, so two people applying at once can't interleave, and encrypt it, because it contains resource details and sometimes secrets.

Terraform, OpenTofu, and the fork in the road

In 2023 HashiCorp relicensed Terraform under the Business Source License, and the community forked the last open version as OpenTofu, now under the Linux Foundation. In practice the two remain compatible: the same HCL, the same providers, the same state format, and most configurations move without edits. OpenTofu has shipped features Terraform hasn't, notably state encryption; Terraform has HashiCorp's platform around it, and IBM behind it since the 2025 acquisition. Either is a reasonable choice. What matters is that you pick one, keep your configuration portable, and don't spend a quarter on the question.

terraform {
  required_version = ">= 1.11"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.60"
    }
  }
  backend "s3" {
    bucket       = "acme-tfstate-production"
    key          = "invoicing/storage.tfstate"
    region       = "eu-west-1"
    encrypt      = true
    use_lockfile = true # state locking in S3 itself, no separate table
  }
}

variable "environment" {
  type = string
  validation {
    condition     = contains(["staging", "production"], var.environment)
    error_message = "environment must be staging or production."
  }
}

resource "aws_s3_bucket" "pdfs" {
  bucket = "invoicing-pdfs-${var.environment}"

  # Renaming this resource in code must never delete the customers' invoices.
  lifecycle {
    prevent_destroy = true
  }
}

resource "aws_s3_bucket_versioning" "pdfs" {
  bucket = aws_s3_bucket.pdfs.id
  versioning_configuration {
    status = "Enabled" # a deletion by our own code is recoverable
  }
}

resource "aws_s3_bucket_public_access_block" "pdfs" {
  bucket                  = aws_s3_bucket.pdfs.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_lifecycle_configuration" "pdfs" {
  bucket = aws_s3_bucket.pdfs.id

  rule {
    id     = "cold-storage-after-90-days"
    status = "Enabled"
    filter {
      prefix = "tenants/"
    }
    transition {
      days          = 90
      storage_class = "STANDARD_IA"
    }
  }
}

output "pdf_bucket" {
  value = aws_s3_bucket.pdfs.bucket
}

Four habits are visible there. The provider version is pinned, so a fresh init next month doesn't bring a new major version. The backend is encrypted and locked. The variable validates itself, so a typo fails at plan time rather than creating invoicing-pdfs-prodction. And the bucket that holds customers' invoices carries prevent_destroy, because the most expensive infrastructure mistakes are deletions that the plan described accurately and nobody read.

The plan is the review

A plan is a diff against reality, and it deserves the attention you give code. Run it on every pull request, post it in the review, and apply only from the main branch after merge. Three things to look for every time:

• Anything that says forces replacement on a resource that holds state. Replacing a database or a volume means creating a new one and deleting the old one, with your data inside it.
• Destroys you didn't intend. Renaming a resource in code, or moving it between modules, reads as "destroy that, create this" unless you tell the tool otherwise with a moved block or a state move.
• A plan that is much larger than your change, which usually means drift: someone changed something by hand, and your apply is about to revert it, or the provider has changed its defaults.

Then automate the parts a tired reviewer will miss: policy-as-code checks in the pipeline that reject public buckets, unencrypted volumes, security groups open to the world, or any plan containing a destroy of a protected resource.

Drift, and where it comes from

Drift is the difference between what your code says and what exists. It appears because someone fixed something during an incident at 3am, which is legitimate, or because a console click was easier, which is not. Detect it with a scheduled plan that alerts when the diff is non-empty, then close the loop: either codify the change, or revert it. Drift that nobody reconciles slowly turns infrastructure as code back into documentation.

Structure, blast radius and the rest of the toolchain

• Split state by environment and by component. One state file for everything means one lock, one blast radius and a twenty-minute plan. Separate production from staging entirely, ideally in different accounts.
• Modules for repeated shapes, kept small and versioned. A module that takes thirty variables is a program with a bad interface.
• Immutable where possible. Replace machines rather than mutating them: bake an image, roll it out, delete the old one. Configuration management such as Ansible is for what you can't replace.
• GitOps for clusters. Argo CD or Flux continuously reconcile a Kubernetes cluster against a repository, which is the same idea applied to a system that already reconciles: the repository becomes the desired state, and drift is corrected rather than reported.
• Secrets stay out. Reference a secrets manager rather than committing values, and remember that anything the tool reads can land in state, which is why the state is encrypted and access to it is restricted (chapter 6).

  # aws_db_instance.primary must be replaced
-/+ resource "aws_db_instance" "primary" {
      ~ engine_version       = "15.5" -> "16.3" # forces replacement
      ~ id                   = "invoicing-prod" -> (known after apply)
      ~ endpoint             = "invoicing-prod.xxxx.rds.amazonaws.com" -> (known after apply)
        # (23 unchanged attributes hidden)
    }

  # aws_s3_bucket.uploads will be destroyed
  - resource "aws_s3_bucket" "uploads" {
      - bucket = "invoicing-uploads-production" -> null
    }

  # module.network.aws_security_group_rule.db_ingress will be updated in-place
  ~ resource "aws_security_group_rule" "db_ingress" {
      ~ cidr_blocks = ["10.0.0.0/16"] -> ["0.0.0.0/0"]
    }

Plan: 2 to add, 1 to change, 2 to destroy.

The shape of a pipeline

Cheap checks first, so feedback is fast, and one artefact built once and promoted everywhere afterwards (The Engineer's Shipyard, chapter 17). A workable order: format and lint, type check, unit tests, build the artefact, integration tests against real dependencies started as containers, security and dependency scans, publish the artefact with its provenance, then hand over to deployment (chapter 10). Everything before the artefact runs on every pull request; everything after runs on the main branch.

name: ci

on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read          # the default token gets nothing else unless a job asks

concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true          # a new push cancels the old run

jobs:
  test:
    runs-on: ubuntu-24.04
    timeout-minutes: 15             # a hung job fails instead of burning an hour
    services:
      postgres:                     # a real database, not a mock
        image: postgres:17@sha256:aaaa...
        env: { POSTGRES_PASSWORD: test }
        options: --health-cmd pg_isready --health-interval 5s --health-retries 10
        ports: ["5432:5432"]
    steps:
      - uses: actions/checkout@08c6903  # v5.0.1 - third-party actions pinned by commit
      - uses: actions/setup-python@a26af6..  # v6.0.0
        with:
          python-version: "3.13"
          cache: pip
      - run: pip install -r requirements-dev.txt
      - run: ruff check .
      - run: mypy src
      - run: pytest -q --maxfail=1 --durations=10
        env:
          DATABASE_URL: postgres://postgres:test@localhost:5432/postgres

  image:
    needs: test
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-24.04
    timeout-minutes: 20
    permissions:
      contents: read
      id-token: write               # OIDC: a short-lived cloud token, no stored keys
    steps:
      - uses: actions/checkout@08c6903  # v5.0.1
      - uses: docker/setup-buildx-action@e468171  # v3.11.1
      - uses: aws-actions/configure-aws-credentials@f503a18  # v5.0.0
        with:
          role-to-assume: arn:aws:iam::123456789012:role/ci-push-images
          aws-region: eu-west-1
      - name: Build, attest and push one artefact
        run: |
          docker buildx build \
            --sbom=true --provenance=true \
            --cache-from type=registry,ref=$REGISTRY/invoicing/api:cache \
            --cache-to type=registry,ref=$REGISTRY/invoicing/api:cache,mode=max \
            --tag $REGISTRY/invoicing/api:${{ github.sha }} \
            --push .
        env:
          REGISTRY: 123456789012.dkr.ecr.eu-west-1.amazonaws.com

The security-relevant lines are easy to miss. permissions: contents: read at the top means the automatic token can do nothing else unless a job asks; the image job asks for id-token: write so it can exchange an OIDC token for a cloud role, which removes the long-lived access key that would otherwise sit in a CI variable forever (chapter 7). Third-party actions are pinned to a commit, not a tag, because a tag can be moved by whoever owns the repository, and an action runs with access to your build. Every job has a timeout, because a hung job is a wasted hour and sometimes a wasted budget.

Speed is a feature

A pipeline you wait for is a pipeline people work around: they batch changes, skip runs, and merge on red. Aim for pull-request feedback in about ten minutes, and treat anything longer as a bug with an owner. The usual wins, in order of payoff:

• Cache dependencies and build layers. Downloading the world on every run is the most common single cost.
• Run independent things in parallel, and shard the test suite across several machines.
• Fail fast on the cheapest signal, and cancel superseded runs, as the concurrency block above does.
• Only build once. If the image is built again for the deploy, you are testing one artefact and shipping another.
• Keep integration tests honest but few. They're the slowest and the most valuable; spend the time deliberately.

Flaky tests are a reliability problem

A test that fails one run in a thousand sounds harmless until you multiply. Five hundred such tests turn a green suite into a coin flip: the probability that all of them pass is 0.999⁵⁰⁰ ≈ 61%, so two runs in five are red for no reason. Two thousand of them, and 14% of runs pass.

The damage isn't the lost time, it's the lost signal: once "just rerun it" becomes normal, a real failure is indistinguishable from noise, and the pipeline has stopped answering the only question it exists to answer. Treat flakiness as a defect: detect it by rerunning failures automatically and recording which tests pass on retry; quarantine the worst offenders out of the blocking suite with an owner and a deadline; and fix the causes, which are nearly always shared state, real clocks, real networks, or a dependency on test ordering.

The pipeline is a production system

It has credentials, it runs code, and it publishes artefacts that go straight to your users, which makes it a target. The defaults worth having: least-privilege tokens; no secrets available to workflows triggered by pull requests from forks; third-party actions pinned by commit; lockfiles committed and dependency updates reviewed; artefacts signed with an SBOM and provenance attached (chapter 4); and a protected main branch where checks must pass, review is required, and nobody pushes directly.

Measuring delivery

DORA's four keys remain the useful summary: deployment frequency and lead time for changes for throughput, change failure rate and failed deployment recovery time for stability. The finding that made them famous is that these move together: teams that deploy more often also fail less, because small changes are easier to review, test and reverse. Use them to watch your own trend, not to compare teams, and remember that any metric a team is judged on becomes a metric a team can produce.

on: pull_request_target

permissions: write-all

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.sha }}
      - uses: some-org/deploy-action@main
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

Three ways to replace a running version

In practice most teams run rolling updates by default (chapter 5), blue-green where a switch must be instant and capacity is cheap, and canaries for the changes that scare them. Feature flags are the finest grain of all, letting the deploy and the release be separate events, which is the subject of The Engineer's Shipyard, chapter 17.

What a canary can actually see

Canary analysis compares the new version against a baseline on four things: error rate, latency percentiles, saturation, and at least one business metric such as invoices created a minute. But detection is a statistics problem, and the arithmetic is sobering. To notice an error rate rising from 0.1% to 1% with reasonable confidence you need on the order of a thousand requests through the canary; to notice a rise to 0.2%, over twenty thousand.

So a ten-minute canary on a small service can only catch a disaster, not a regression, and that's fine as long as you know it: the canary catches the catastrophic, the error budget and alerting catch the subtle (chapter 15). Two details make the comparison honest: compare against a baseline of the same age, ideally a freshly started set of old-version instances rather than the long-warm fleet, since cold caches and cold runtimes make any new deployment look slower; and give the canary representative traffic, because 5% routed by user ID may be 5% of a very unrepresentative kind.

Rollback is the feature

The single most valuable property of a deployment system is that you can undo it quickly, without thinking. In an incident, roll back first and diagnose afterwards (chapter 16): you cannot debug faster than your users are suffering. That means rollback must be one command or one button, must be practised, and must not depend on the thing that's broken.

The usual thing that makes rollback impossible is the database. A migration that drops a column, renames one, or writes data in a new format can't be reversed by redeploying the old binary, which no longer understands the data. The discipline is expand and contract: deploy the schema change that both versions tolerate, deploy the code, then remove the old shape in a later release, once you're sure. Never ship a destructive migration in the same release as the code that depends on it.

Make the platform do the safety

• Add capacity before removing it, with maxUnavailable: 0 and a real readiness probe (chapter 5).
• Fail the rollout automatically when it stops making progress, and have the pipeline watch rollout status rather than declaring success on apply.
• Gate on signals, not on time. An automated canary analysis that checks error rate and latency against the baseline for a fixed window is worth more than a human deciding it "looks fine".
• Deploy one thing at a time, so a bad change is identifiable. Batching five changes into one release means bisecting in production.
• Keep the previous artefact ready to deploy, by digest.

As for deploying on a Friday: the day isn't the risk. The risk is deploying when you can't observe the result and can't reverse it, which is equally true on Tuesday morning. Teams that deploy rarely have riskier deploys, because each one carries more change, which is the mechanism behind DORA's finding that frequency and stability rise together.

What to measure

Two checklists cover most of it. For every service, RED: request rate, error rate, and duration as a distribution. For every resource, USE from chapter 2: utilisation, saturation, errors. Google's four golden signals say the same thing in different words: latency, traffic, errors, saturation. Add one or two numbers that describe the product working, such as invoices created a minute, because those are the only metrics that notice when the system is up and useless.

from prometheus_client import Counter, Histogram

# Labels are dimensions you will group by. Never put anything unbounded in one:
# no user ids, no request ids, no raw URLs. Route templates, not paths.
REQUESTS = Counter(
    "http_requests_total", "HTTP requests",
    ["method", "route", "status"],
)
LATENCY = Histogram(
    "http_request_duration_seconds", "Request duration",
    ["method", "route"],
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0),
)


def record(method, route, status, seconds):
    REQUESTS.labels(method=method, route=route, status=str(status)).inc()
    LATENCY.labels(method=method, route=route).observe(seconds)

Scraped, that produces a handful of series per endpoint: two counters and one histogram's worth of buckets.

http_requests_total{method="GET",route="/invoices/{id}",status="200"} 993.0
http_requests_total{method="GET",route="/invoices/{id}",status="500"} 7.0
http_request_duration_seconds_bucket{le="0.005",method="GET",route="/invoices/{id}"} 0.0
http_request_duration_seconds_bucket{le="0.01",method="GET",route="/invoices/{id}"} 23.0
http_request_duration_seconds_bucket{le="0.025",method="GET",route="/invoices/{id}"} 420.0
http_request_duration_seconds_bucket{le="0.05",method="GET",route="/invoices/{id}"} 882.0
...
http_request_duration_seconds_bucket{le="+Inf",method="GET",route="/invoices/{id}"} 1000.0
http_request_duration_seconds_count{method="GET",route="/invoices/{id}"} 1000.0
http_request_duration_seconds_sum{method="GET",route="/invoices/{id}"} 55.9

Counters only ever increase, which is what makes them reliable across restarts: the query computes a rate over them, and a reset to zero is detected rather than counted as a huge negative. Gauges are for things that go up and down, such as queue depth or connections in use. Histograms are for distributions, which is nearly always what you want for latency and size.

Cardinality is the budget

A metric's cost is its number of series, and that's the product of its label values. One request counter with 10 routes, 4 methods and 12 status codes is 480 series per instance, and 24,000 across 50 instances: entirely normal. Add a customer_id label with 5,000 customers and it becomes 120 million, which will take down your monitoring before it tells you anything about your service.

So: labels must have small, bounded sets of values. Route templates, not URLs with identifiers in them. Status classes or codes, not messages. Never user, request, trace, session or order identifiers, email addresses, or anything a user can influence. That detail belongs in logs and traces, which are built for high cardinality (chapters 12 and 13). When you need per-customer numbers, aggregate them in a separate, deliberately small set of series for your biggest few tenants, or compute them from logs.

Histograms, and the truth about your p99

A histogram counts how many observations fell into each bucket. Percentiles are then estimated by finding the bucket the quantile lands in and interpolating across it, which means your p99 is only as good as your bucket boundaries. Ten thousand requests, mostly around 30 ms with 2% of them between 1.5 and 3 seconds, give a true p99 of 2.04 s. With well-chosen buckets the estimate is 2.01 s. With four round buckets at 10 ms, 100 ms, 1 s and 10 s, the same data gives 4.9 s.

Prometheus 3.0, released at the end of 2024, adds native histograms, which choose exponential buckets automatically at a resolution you pick, and largely remove this problem along with the guesswork. The same release ingests OpenTelemetry metrics directly and accepts UTF-8 metric names. If you are starting now, start there.

Querying without lying to yourself

Four patterns cover most dashboards. rate(http_requests_total[5m]) for throughput, always summed after the rate, never before. sum by (route) (rate(...)) to group. histogram_quantile(0.99, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))) for a latency percentile, which works precisely because histogram buckets can be added across instances even though percentiles cannot (chapter 3). And an error ratio as a division of two rates, which is what your objective will be built from.

Expensive queries that dashboards and alerts both need belong in recording rules, evaluated once and stored, so an incident doesn't begin with slow dashboards.

Events, not sentences

A log line is a structured event that happens to be readable, not a sentence with values glued into it. "Processed invoice 8812 for acme in 142ms" is unsearchable at scale; the same event as JSON fields can be filtered, grouped, counted and correlated. Every line should carry the same core context, and adding that context by hand at every call site guarantees it will be missing exactly where you need it.

import contextvars
import json
import logging
import time
import zlib

# Set once per request; every line logged while handling it carries these fields.
context = contextvars.ContextVar("log_context", default={})

SENSITIVE = ("password", "token", "secret", "authorization", "card", "api_key")


def _redact(key, value):
    return "***" if any(word in key.lower() for word in SENSITIVE) else value


class JsonFormatter(logging.Formatter):
    def format(self, record):
        event = {
            "ts": time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(record.created)) + ".%03dZ" % (record.msecs),
            "level": record.levelname.lower(),
            "logger": record.name,
            "message": record.getMessage(),
        }
        event.update(context.get())
        event.update(getattr(record, "fields", {}))
        if record.exc_info:
            event["error_type"] = record.exc_info[0].__name__
            event["stack"] = self.formatException(record.exc_info)
        return json.dumps({k: _redact(k, v) for k, v in event.items()}, default=str)


def log(logger, level, message, **fields):
    """log(logger, logging.INFO, "invoice sent", invoice_id=..., channel="email")"""
    logger.log(level, message, extra={"fields": fields})


def keep(trace_id: str, rate: float) -> bool:
    """Deterministic sampling: every service keeps the same traces, so they stay complete."""
    return zlib.crc32(trace_id.encode()) % 10_000 < rate * 10_000

Set the context once when the request arrives, and every line logged anywhere under it, including from libraries and background tasks in the same task context, carries the trace ID, tenant and version:

{"ts": "2026-09-25T17:35:45.483Z", "level": "info", "logger": "invoicing", "message": "invoice sent",
 "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "tenant": "acme", "version": "2.7.1",
 "invoice_id": "inv_8812", "channel": "email", "duration_ms": 142}
{"ts": "2026-09-25T17:35:45.483Z", "level": "warning", "logger": "invoicing", "message": "provider slow",
 "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "tenant": "acme", "version": "2.7.1",
 "provider": "postmark", "api_key": "***", "duration_ms": 2400}

Note the redaction: a field whose name looks sensitive never reaches the log store, whoever wrote the call. That's the only kind of rule that holds, because the alternative depends on every engineer remembering, forever.

Correlation is the whole point

One field matters more than the rest: the trace ID, shared with the tracing system (chapter 13) and propagated to every service the request touches. With it, a customer complaint becomes one search that returns every line from every service for that request, in order. Without it, an investigation is a series of guesses about timestamps. Add the tenant, the release version, and the identifiers a human would ask about, such as the invoice ID, and stop: fields are cheap, but a schema nobody agreed on is not.

Levels, and saying something worth reading

• Error means a human should look at this, eventually. If nobody would act, it is not an error.
• Warning means something recovered but is worth knowing: a retry, a fallback, a slow dependency.
• Info records the decisions a system made: sent, rejected, refunded, retried, throttled.
• Debug is for development and for temporary investigation, off in production by default.

Log an event once, where the decision was made, rather than at every layer it passes through. Log the reason, not just the outcome: "rejected" is useless, {"decision": "rejected", "reason": "tax_id_invalid"} is a dashboard. And never log secrets, tokens, card numbers or personal data; retention and access control for logs are almost always weaker than for your database, which makes logs a favourite place for data to leak out of (stage 13).

What it costs, and what to do about it

Logging is priced by volume, and volume grows with traffic. Four lines per request at 500 bytes, at 2,000 requests a second, is 4 MB a second: 346 GB a day, 10 TB a month, which at hosted prices is a serious line item and at self-hosted scale is a cluster someone has to run.

Three levers, in order of effect:

• Sample successes, keep failures. Deterministic sampling on the trace ID, as in keep() above, means every service makes the same decision, so a kept trace is complete rather than full of holes.
• Drop debug in production, and make turning it on for one service, or one tenant, a runtime switch (chapter 6).
• Tier retention: days in a searchable store, months in compressed object storage you can restore from when an auditor asks.

A policy of "keep one summary line per successful request, sample the detail at 5%, keep everything for errors and slow requests" takes that 346 GB a day down to about 100 GB, a 70% cut, while making incident investigation better, because what remains is signal.

Getting them out of the process

Write to standard output and let the platform collect it (chapter 1). A collector such as the OpenTelemetry Collector, Fluent Bit or Vector adds metadata, batches, and forwards to a store. Don't write log files inside containers, don't make the application talk to the log store directly, and remember that this path is best-effort: logs get dropped under load, so nothing that must be exact, such as billing or audit, may depend on them.

Audit logs are a different system with different rules: who did what to which record, written durably as part of the transaction, append-only, retained for years, and readable by people who are not engineers.

INFO  Processing...
ERROR Exception occurred
INFO  User admin@acme.com paid with card 4111111111111111, token=eyJhbGci...
WARN  Retrying (attempt 3)
DEBUG {"body": "<the entire 40KB request>"}

Spans, traces and the header that ties them together

A span is one operation with a start, a duration, attributes and a parent. A trace is the tree of spans that share a trace ID. When a service calls another, it sends the current trace and span IDs in a header, and the receiver makes its spans children of that one. The W3C standard header looks like this:

traceparent: 00-b6a6820054606aad82c4b3f5c2e55b1a-1d3858ed71703947-03
             ^  ^                                ^                ^
             |  trace id (16 bytes)              parent span id   flags: sampled
             version

That single header is what makes a distributed system explicable. The same trace ID goes into your logs (chapter 12), so one identifier connects every signal you have.

Instrumenting

OpenTelemetry is the standard: one API and SDK per language, plus a collector that receives, processes and forwards. Traces, metrics and logs are stable across the major languages; continuous profiling entered public alpha in 2026, the year the project graduated from the CNCF. Use automatic instrumentation for your framework, HTTP client and database driver, then add manual spans for the steps your business cares about.

import time
from opentelemetry import trace
from opentelemetry.propagate import inject
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter

trace.set_tracer_provider(TracerProvider(
    resource=Resource.create({"service.name": "invoicing-api", "service.version": "2.7.1"})))
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
tracer = trace.get_tracer(__name__)


def get_invoice(invoice_id, tenant):
    with tracer.start_as_current_span("GET /invoices/{id}") as span:
        span.set_attribute("http.route", "/invoices/{id}")   # bounded, like a metric label
        span.set_attribute("tenant.id", tenant)              # high cardinality is fine here
        span.set_attribute("invoice.id", invoice_id)
        check_permissions(tenant)
        invoice = load_invoice(invoice_id)
        render_pdf(invoice)
        return invoice


def check_permissions(tenant):
    with tracer.start_as_current_span("authz.check"):
        time.sleep(0.004)


def load_invoice(invoice_id):
    with tracer.start_as_current_span("db.query invoices") as span:
        span.set_attribute("db.statement", "SELECT * FROM invoices WHERE id = $1")
        time.sleep(0.011)
        lines = [f"line_{i}" for i in range(5)]
    for line in lines:                                        # one query per line: the N+1
        with tracer.start_as_current_span("db.query invoice_lines") as span:
            span.set_attribute("db.statement", "SELECT * FROM invoice_lines WHERE id = $1")
            time.sleep(0.009)
    return {"id": invoice_id, "lines": lines}


def render_pdf(invoice):
    with tracer.start_as_current_span("pdf.render") as span:
        headers = {}
        inject(headers)            # W3C traceparent, so the next service joins this trace
        span.set_attribute("http.request.header.traceparent", headers["traceparent"])
        time.sleep(0.030)

Attributes here can be as specific as you like: tenant, invoice, user, query. That's the complement to chapter 11: metrics must keep cardinality low, and traces exist precisely to carry the high-cardinality detail. Follow the semantic conventions for the standard names, such as http.route and db.statement, so tools understand your spans without configuration.

Reading a waterfall

Four shapes are worth recognising at a glance. A staircase of identical spans is an N+1 query or a loop of remote calls. One very long child is a slow dependency, and the question becomes whether it's slow for everyone. A gap between spans is time nobody instrumented: queueing, garbage collection, a lock, or a service that isn't traced. And a wide parent with narrow children means the time is in your own code, which is where a profiler comes in (chapter 2).

Sampling

Tracing every request is affordable at small scale and ruinous at large scale, so you sample. Head sampling decides at the first service and propagates the decision in that 03 flag, which keeps traces complete but is blind: you throw away 99% of requests before knowing which ones were interesting. Tail sampling, done in the collector, buffers the spans of a trace and decides once it's finished, so you can keep every trace that errored, every trace slower than your objective, and a small percentage of the rest. That's the policy most teams end up with, and it's worth the collector.

Whatever you choose, make the logging sampler agree with it, and keep exemplars: metric buckets that carry a few example trace IDs, so a spike on a dashboard is one click from a trace that shows why.

The gap that ruins traces

Traces break at asynchronous boundaries. A request publishes a message; the consumer runs five seconds later in another process and starts a brand-new trace, and the story ends exactly where the interesting part began. The fix is to carry the trace context in the message itself, in a header field, and have the consumer start its span as a child of, or a link to, the producer's. Scheduled jobs and retries deserve the same treatment: a background task with no trace context is a black box the moment it matters.

Dashboards that answer questions

Start with one page per service that answers "is it healthy?" in ten seconds: request rate, error ratio, latency percentiles, saturation, and the one business metric that says the product is working. Then a second page for "what changed?": deploys and configuration changes as annotations on the same time axis, plus the health of the dependencies. Everything else belongs in drill-downs that you open when the first two pages point somewhere.

Details that make the difference: the same layout on every service's page, so people can read one they've never seen; consistent time ranges across panels; percentiles rather than averages (chapter 3); ratios rather than counts; and a maximum across instances next to the aggregate, so a single sick pod is visible. A forty-panel wall built by accretion is not a dashboard, it's a museum.

Alert on symptoms, not causes

The rule from Google's SRE practice is that pages should be about things users can feel: requests failing, requests too slow, work not being done. Causes such as high CPU, a full disk or a restarting pod belong on dashboards and in tickets, because they may or may not matter, and because a page for each cause means several pages for one incident. Three questions for every alert that wakes someone: is it real, is it urgent (does it need action now rather than tomorrow), and is it actionable (does the person know what to do)? If any answer is no, it's a ticket or a dashboard panel.

Burn-rate alerting

The best-behaved symptom alert is built on the error budget from chapter 15. If your objective is 99.9% over 30 days, the budget is 43.2 minutes of failure, and what matters is how fast you're spending it. A burn rate of 1 spends the budget exactly over the window; a burn rate of 14.4 spends 2% of a 30-day budget in an hour. Google's recommended starting point pairs a long window with a short one, so alerts fire fast and also stop fast:

The second window is the part people leave out, and it's what makes the alert usable.

groups:
  - name: invoicing-api-slis
    interval: 30s
    rules:
      # One definition of "bad", recorded once and used by every alert and dashboard.
      - record: job:slo_errors:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{job="invoicing-api",status=~"5.."}[5m]))
            / sum by (job) (rate(http_requests_total{job="invoicing-api"}[5m]))
      - record: job:slo_errors:ratio_rate1h
        expr: |
          sum by (job) (rate(http_requests_total{job="invoicing-api",status=~"5.."}[1h]))
            / sum by (job) (rate(http_requests_total{job="invoicing-api"}[1h]))
      - record: job:slo_errors:ratio_rate6h
        expr: |
          sum by (job) (rate(http_requests_total{job="invoicing-api",status=~"5.."}[6h]))
            / sum by (job) (rate(http_requests_total{job="invoicing-api"}[6h]))

  - name: invoicing-api-burn-rate
    rules:
      # 99.9% availability over 30 days. 14.4x burn for an hour spends 2% of the budget.
      - alert: InvoicingApiBudgetBurningFast
        expr: |
          job:slo_errors:ratio_rate1h > (14.4 * 0.001)
            and job:slo_errors:ratio_rate5m > (14.4 * 0.001)
        for: 2m
        labels:
          severity: page
          service: invoicing-api
        annotations:
          summary: Invoicing API is spending its error budget 14x too fast
          description: >-
            At this rate 2% of the 30-day budget goes in an hour. Mitigate first:
            check the last deploy and config change, then roll back.
          runbook: https://runbooks.internal/invoicing-api/budget-burn

      # 6x burn over six hours spends 5% of the budget: slower, still worth waking someone.
      - alert: InvoicingApiBudgetBurningSteadily
        expr: |
          job:slo_errors:ratio_rate6h > (6 * 0.001)
            and job:slo_errors:ratio_rate1h > (6 * 0.001)
        for: 15m
        labels:
          severity: page
          service: invoicing-api
        annotations:
          summary: Invoicing API has been failing steadily for hours
          runbook: https://runbooks.internal/invoicing-api/budget-burn

Note what each part buys. The recording rules give one definition of "bad" that dashboards and alerts share, so nobody argues during an incident about whether 499s count. The for clause absorbs single scrapes. The severity label routes the alert. And every alert carries a runbook link, because the person it wakes may have joined last month.

Keeping the pager honest

• Every page needs an owner and a runbook. An alert nobody owns is deleted, not inherited.
• Budget the pages. A widely used rule of thumb is no more than about two pages per on-call shift; beyond that, people stop reading them properly, and burnout follows (chapter 16).
• Review alerts monthly. For each page: was it real, was it actionable, did it need to be at that hour? Delete or downgrade the failures.
• Group and inhibit. One cause should produce one page, not twelve: group by service, and suppress dependent alerts when the upstream one is firing.
• Silence deliberately, with an expiry, during known maintenance. A permanent silence is a deleted alert with extra steps.
• Alert on absence too: no requests, no jobs run, no data received. Silence is the failure mode monitoring misses most often.

Indicator, objective, budget

A service level indicator is a ratio: good events divided by valid events, measured as close to the user as you can get. A service level objective is a target for that ratio over a window, such as 99.9% over 30 days. The error budget is what's left: the amount of failure you have explicitly decided to tolerate, and therefore to spend.

The budget is the part that changes behaviour. It converts an argument about whether to ship into arithmetic, and it gives reliability work a moment when it automatically outranks features.

Read that table before promising anything. At 99.99%, a single four-minute deploy that goes wrong spends the entire month, which means no manual response is fast enough: everything must be automatic. Each additional nine costs roughly an order of magnitude more engineering, so the question is never "how reliable can we be" but "how reliable do our users need us to be, and what will we give up for it".

Choosing indicators that mean something

Pick two to four per user journey, not per service:

• Availability: the proportion of valid requests that got a successful response. Decide up front what counts: a 400 caused by the client is usually not your failure; a 429 you issued under load probably is.
• Latency: the proportion of requests faster than a threshold. Note the shape: not "p99 under 300 ms" but "99% of requests under 300 ms", which is the same number stated so that it can be summed over a month.
• Freshness for pipelines and derived data: the proportion of time the data is newer than some age.
• Correctness where it matters more than speed: invoices with correct totals, payments reconciled.

Measure where the user is. A server-side timer excludes queueing, the load balancer, DNS and the network, which is where slow requests are born (chapter 3). Use the load balancer's numbers, or real user measurements, and use synthetic checks for services too quiet for statistics to work.

Spending the budget

With a 99.9% objective over 30 days, you have 43.2 minutes. One 22-minute outage spends 51% of it. A partial failure counts proportionally: 40% of requests failing for 50 minutes spends 46%. Counting requests rather than minutes gives the same answer and handles partial failure naturally: at 2,000 requests a second, a month is about 5.2 billion requests, the budget is 5.2 million failures, and that incident used 2.4 million of them.

The policy is what makes it real, and it must be agreed before you need it, with whoever decides priorities:

A budget nobody spends is also information: if you finish every month at 99.99% against a 99.9% objective, you are buying reliability nobody asked for, and you could be shipping faster or spending less (chapter 19).

What you can promise

Your objective is bounded by what you depend on. An API that calls three services in sequence, each with a 99.95% objective, cannot promise better than about 99.85% before its own code fails at all (The Engineer's Fleet, chapter 1). Either reduce the dependencies on the critical path, make them optional with fallbacks, or promise less. Setting an objective higher than the arithmetic allows is a way of agreeing to fail.

Mitigate first

The instinct of every engineer is to understand the problem. During an incident that instinct is wrong. Your first job is to stop the damage, and the moves that do it rarely require knowing why: roll back the last deploy, revert the last configuration change, fail over to another zone, shed load, disable the expensive feature, turn off the batch job that's crushing the database. Understanding can wait for the postmortem, when nobody is losing money.

This has a corollary worth stating plainly: build the mitigations before you need them. A rollback button, a feature flag, a load-shedding switch, a read-only mode, a way to pause consumers. In the middle of an incident you will use the tools that already exist, not the ones you wish existed.

Roles, even in a team of four

The thing that makes incidents go badly is not a lack of technical skill, it's six people investigating the same thing while nobody talks to customers. Name roles out loud, even if one person holds two of them:

• Incident commander: decides, tracks what's being tried, and does not touch a keyboard. Their job is the shape of the response, not the fix.
• Operations: the people making changes, one at a time, saying what they're doing before they do it.
• Communications: updates the status page and the internal channel on a cadence, and is the single voice to the rest of the company.
• Scribe: writes the timeline as it happens. Timestamps, actions, observations. This is worth a whole person, because nobody can reconstruct it afterwards.

The commander's most useful sentences are "what do we know", "what are we trying next", "who is doing it" and "when do we check back".

Severity, declared early

Declare early and downgrade later. The cost of declaring a sev 1 that turns out to be a sev 2 is a few people's attention for twenty minutes; the cost of the reverse is discovering at hour three that nobody told the customers.

The first ten minutes

1. Declare. Open a channel, name it after the incident, state the symptom and the time it started.
2. Take command, or say who has it. Post it in the channel so it's unambiguous.
3. Establish impact: who is affected, how many, since when, and which user journey. This determines everything else.
4. Check what changed: deploys, configuration, feature flags, infrastructure changes, certificate expiries, and the dependencies' status pages. Most incidents are something someone did.
5. Mitigate with the biggest reversible lever you have.
6. Tell people: internal update, and the status page if customers can see it. Say what you know, what you don't, and when you'll next update.

Communicating while it burns

Update on a fixed cadence even when there is nothing new, because silence is read as chaos. Write for customers in terms of what they can and can't do, not components: "invoices created in the last hour may not have been emailed; we are resending them" rather than "the queue consumer is stuck". Use UTC and absolute timestamps everywhere. Keep speculation out of customer-facing messages and put it in the internal channel, where it belongs.

Sustainable on call

An on-call rotation is a load-bearing part of your system, and it fails the same way as any other overloaded component.

• Enough people. A 24/7 rotation needs six to eight, or two teams in different time zones. Four people means one week in four, which people leave over.
• A page budget. Aim for at most a couple of pages per shift, and treat anything above that as a bug in the alerts or in the system (chapter 14).
• Recovery time. Someone up at 3am does not work the next day. This has to be a rule, not a kindness.
• Pay for it, or at least compensate in time. Unpaid on call is a quiet tax on the people least able to refuse it.
• A handover ritual: what fired, what's still broken, what to watch. Five minutes that saves the next person an hour.
• Only page for what a human can fix now. If the answer is "restart it", automate the restart.

New engineers should shadow a rotation before joining it, and every team should run the occasional game day: break something in a controlled way and practise the response, including the parts that are not technical, such as finding the status page password (The Engineer's Fleet, chapter 20).

Blameless means "the system allowed it"

Blameless does not mean nobody is accountable. It means that when someone ran the command, approved the plan or missed the alert, the useful question is not "why were they careless" but "why was it easy to do, and why did nothing catch it". People act reasonably given what they know at the time; if the outcome was bad, the information, the tooling or the defaults were bad.

There is a practical reason as well as a humane one. In a culture where mistakes are punished, the details you need stop arriving: people hesitate to say what they did, near misses are never reported, and your picture of the system becomes a fiction. The test of a blameless culture is simple: does the person who typed the command write the postmortem, and do they still work here?

What it contains

• Impact, in user terms and numbers: who was affected, how many, for how long, what it cost, how much error budget it spent.
• Detection: how you found out, and how long that took. If a customer told you, say so.
• Timeline: facts with timestamps, including the actions you took that didn't help.
• Contributing factors, plural. Complex systems fail because several things line up.
• What went well, which is how good practice gets reinforced, and where you were lucky, which is the most valuable section and the one most often missing.
• Actions with owners, dates and priorities.

Resist "the root cause". Asking why five times produces a single chain that flatters whoever wrote it. The Cloudflare outage of November 2025 was a database permissions change, and a generated file with no size validation, and a hard limit in a proxy that failed closed instead of degrading, and a deployment path for that file that reached the whole network at once (chapter 6). Remove any one and the day is ordinary.

A worked example

# Invoicing API failing for 23 minutes, 14 March 2026

**Impact.** 09:12-09:35 UTC, 62% of invoice-send requests failed (~310,000 requests,
2,400 customers). 4,100 invoices were delayed; all were sent by 10:20. No data lost.
Spent 51% of March's error budget.

**Detection.** The fast burn-rate alert paged at 09:14, two minutes after the first
errors. The first customer email arrived at 09:21.

## Timeline (UTC)

| 09:08 | Release 2.7.0 finishes rolling out (canary: 3 minutes at 5%).
| 09:12 | Error ratio rises from 0.05% to 62%.
| 09:14 | Page. On-call acknowledges, opens #inc-2026-03-14.
| 09:17 | Database and PDF provider both healthy; dashboards show no saturation.
| 09:21 | On-call correlates the start with the 09:08 deploy.
| 09:22 | Rollback to 2.6.4 started.
| 09:26 | Rollback complete; error ratio begins falling.
| 09:35 | Baseline restored. Incident downgraded to sev 2.
| 10:20 | Retry worker has resent all delayed invoices.

## What happened

2.7.0 introduced a connection pool for the PDF service, sized from a new setting,
PDF_POOL_SIZE. The setting existed in staging but not in production. The config
loader fell back to a default of 2, so every instance could make two concurrent PDF
calls; requests queued behind them and timed out.

## Contributing factors

1. A new required setting had a silent default instead of failing at start-up.
2. Staging had the value because it was set by hand in January; it was never in code,
   so "it works in staging" proved nothing.
3. The canary ran 3 minutes at 5% of traffic: ~18,000 requests, below the number
   needed to detect this reliably, and its analysis only checked latency, not errors.
4. Pool saturation was not instrumented, so no dashboard showed the queue building.

## What went well

- The burn-rate alert fired in two minutes, before any customer wrote in.
- Rollback took four minutes and was not debated.
- The retry worker resent every delayed invoice with no manual work.

## Where we were lucky

- It happened at 09:12 on a Tuesday. The same release at 02:00 would have run for
  hours: our slow burn-rate alert would have paged at ~01:30 into the budget, and
  nobody would have correlated it with a deploy from the previous afternoon.

## Actions

| Action | Owner | By | Priority |
|---|---|---|---|
| Required settings fail at start-up, with all problems listed | R. Okafor | 21 Mar | P1 |
| Canary analysis: check error ratio, require a minimum request count | S. Lind | 28 Mar | P1 |
| Instrument pool saturation and alert on queueing | R. Okafor | 28 Mar | P2 |
| Move staging config into the repository; add drift detection | T. Almeida | 11 Apr | P2 |

Actions that actually happen

Three or four actions, each small enough to finish in a fortnight, beat fifteen aspirations. Prefer changes that make the failure impossible or harmless over changes that ask people to be more careful: a validation, a default, a guard rail, a test. "Add more monitoring" is not an action; "alert when the PDF pool has been saturated for five minutes, owned by Rita, by the 28th" is.

Then track them like any other work, and review completion monthly. An organisation that writes excellent postmortems and completes none of the actions has built an expensive storytelling habit. If the same contributing factor appears in three postmortems, that is the finding, and it outranks everything else on the roadmap.

The meeting, and the audience

Hold the review within a few days, while memory is fresh: thirty minutes, with the people involved and anyone who wants to learn. The author walks the timeline, the group challenges the contributing factors and the actions, and nobody relitigates decisions with information that wasn't available at the time. Publish it where the whole engineering organisation can read it, because the second most valuable reader is someone on another team who has the same latent problem.

Customer-facing write-ups are a different document with the same facts: what happened, what it meant for them, what you're changing, no jargon and no blame shifting. Done well, they build more trust than the outage cost; the good public postmortems are worth studying for that reason alone.

Two numbers, agreed with the business

Recovery point objective is how much data you can afford to lose: the gap between the last recoverable moment and the failure. Recovery time objective is how long you can afford to be down. Both are business decisions with engineering price tags, and both are properties of the restore, not of the backup job.

What restores actually cost

Do the arithmetic before you promise anything. Copying a 2 TB database back at 250 MB/s takes 2.2 hours before anything else happens; add index rebuilds, replaying the write-ahead log since the base backup, verification, and reconnecting the application, and five to six hours is a realistic figure. If your stated RTO is four hours, you do not have a four-hour RTO, you have a wish. Either shorten the restore, by keeping a warm standby or more frequent base backups, or change the promise.

The same arithmetic tells you which technique to use. Nightly logical dumps give an RPO of up to a day, which is fine for a reporting database and catastrophic for payments. Continuous archiving with point-in-time recovery gives an RPO of seconds to minutes and lets you restore to just before the bad migration ran, which is the failure you'll actually have.

Replicas are not backups

A replica copies your mistakes faithfully and immediately: the DELETE without a WHERE clause arrives at the standby in milliseconds. Replication protects against a machine dying. Backups protect against everything else, including your own code, your own hands, and an attacker who now has your credentials.

• Keep copies somewhere a compromise of production cannot reach: a different account or project, with different credentials, and object-lock immutability so nothing, including an administrator, can delete them before their retention expires.
• Back up more than the database: object storage (enable versioning), the encryption keys without which the backup is noise, the infrastructure code and state (chapter 8), secrets, DNS, and any data held only by a third party.
• Turn on soft deletes and versioning wherever they exist; most "restore" requests are one table or one file, and these turn a three-hour operation into a two-minute one.
• Check that backups happen, and that they contain something: alert on backup age and size. A backup job that has silently failed for three weeks is a standard ingredient of a bad month.

Drill it

Put a restore drill in the calendar, quarterly, and treat the stopwatch as the deliverable. Restore the production database into an isolated environment, run your verification queries, record how long each phase took, and write down what was missing. Teams that do this routinely discover, in order: that the runbook is out of date, that the restore needs a permission nobody has, that the encryption key is in the environment being restored, and that the restore takes three times as long as anyone thought.

A useful side effect: if you build staging or performance-test environments from the restore path, it gets exercised every week without a ceremony. The best-tested recovery is one that something else depends on.

How much disaster recovery to buy

Most products should be honest about sitting in the first two rows, and spend the difference on the failures that are far more likely: a bad deploy, a bad migration, a deleted bucket, a corrupted table. Multi-region active-active protects against the rarest scenario while adding daily complexity (The Engineer's Fleet, chapters 4 and 7).

Cost per invoice, not cost per month

A total that grows is not automatically a problem; a total that grows faster than the business is. Pick the unit your product is measured in — invoices sent, orders, active tenants, gigabytes processed — and track cost per unit every month. A cost that scales with usage is an input to pricing. A cost that scales with time, or with the number of environments, or with how long ago someone left the company, is waste.

Industry surveys put wasted cloud spend at roughly a quarter to a third of the bill, and Kubernetes clusters routinely run at around 40% utilisation against what workloads have reserved. That is not a moral failing; it is what happens when requests are set by copy-paste and nobody is measured on it.

Read the bill like a stack trace

Here is a month for the invoicing product, sending two million invoices:

Applying the obvious fixes — 35% off compute from commitments and spot, 70% off logs, half the storage tiered, 60% off network, non-production stopped at night, orphans deleted — takes $27,100 a month to about $15,300: a 44% cut, and the cost per invoice falls from 1.35 cents to 0.76. None of it required a negotiation, and none of it made the product less reliable.

Two of those lines deserve emphasis, because they surprise people. Observability often costs more than the database, and grows with traffic and with every new label. And non-production environments quietly cost as much as a small production, because nobody turns them off.

Where the money actually hides

• Reserved but unused capacity. A pod requesting 4 GB and using 600 MB reserves the 4 GB (chapter 5). Right-size from measured usage, and use the autoscaler's recommendations rather than intuition.
• Idle time. Development clusters at 3am, CI runners waiting, instances kept for a spike that happens twice a year.
• Data transfer. NAT gateways, cross-zone chatter, egress to the internet, and traffic to managed services over public paths (chapter 7).
• Storage nobody owns: old snapshots, unattached volumes, logs with infinite retention, and the copy of the production database someone made in 2024.
• Per-request pricing on managed services, which is cheap until a retry loop turns it into the largest line on the bill.
• Engineer time, which is more expensive than nearly all of the above, and is why "we could save $200 a month by self-hosting this" is usually the wrong trade.

Buying efficiency, not fragility

Every saving has a reliability price, and the point of the earlier chapters is to know which ones are worth paying. Running at 90% utilisation saves compute and costs you the latency curve from chapter 3. Removing the standby saves a database and costs you the failover in chapter 18. Deleting the third zone saves cross-zone traffic and costs you the failure domain in chapter 7. Spot instances are free money for queue workers and a bad idea for your only stateful node.

The discipline is to make the trade explicitly and write it down, ideally against the error budget: if a saving would predictably spend more budget than you have (chapter 15), it isn't a saving.

Making it somebody's job

Costs drift upwards unless someone watches, and the mechanisms are boring: tag everything by team and service so the bill can be split; put a per-service cost panel next to the reliability dashboards; alert on anomalies, because a 40% jump overnight is usually a bug, a runaway retry loop or a leaked credential, not growth; review the top five lines monthly; and give teams a budget they own rather than a rule they resent. Engineers respond to a number they can see.

Toil, and why it grows

Google's SRE practice defines toil precisely: work that is manual, repetitive, automatable, tactical, without enduring value, and that grows at least linearly with the size of the service. Restarting a stuck worker is toil. Copying a report by hand every Monday is toil. Approving the same routine change is toil. It is not the same as overhead, and it is not the same as hard work; the distinguishing feature is that it scales with the system, so a team whose toil is unchecked eventually spends all its time keeping yesterday running.

The rule of thumb is to keep it under half of the team's time, and the practice is dull: count it, in a shared list, for a month. Then automate the top item, and only the top item, each month. An alert whose answer is always the same command is the easiest place to start, because it's the one costing you sleep (chapter 16).

Ownership, written down

Every service needs a team whose name is on it, and a catalogue entry that anyone can find in the middle of the night: what it does, who owns it, where its dashboards and runbooks are, what it depends on, what depends on it, and where its code and deploy pipeline live. Unowned services accumulate in every organisation, and they are where incidents go to become long ones.

The same applies to on-call: if the team that builds a service also carries its pager, the feedback loop that makes systems reliable exists. If it doesn't, someone else pays for decisions they can't change, which is both unfair and ineffective.

Runbooks that survive contact

A runbook is not documentation, it's an instruction sheet for someone tired and worried. Write it during or immediately after an incident, while you still remember the commands. Keep each one short: what this alert means, how to confirm it, how to mitigate, how to escalate, and what not to do. Link it from the alert (chapter 14), keep it next to the code so it changes with the code, and test it by having someone who didn't write it follow it during a drill. A runbook nobody has followed is a wiki page.

The production readiness review

Before a new service takes real traffic, walk one checklist. It takes an hour and saves quarters.

• Owned: a named team, an on-call rotation, an entry in the catalogue.
• Observable: RED metrics, structured logs with trace IDs, traces, a dashboard that answers "is it healthy" (chapters 11–14).
• Objectives: at least one SLI and SLO, with an alert on burn rate and a runbook behind it (chapter 15).
• Deployable: one artefact, automated rollout, health-gated, rollback tested (chapters 9 and 10).
• Bounded: resource requests and limits, timeouts on every dependency, retries with budgets, graceful shutdown (chapters 1 and 5).
• Recoverable: backups and a timed restore, or an explicit statement that it holds no state worth keeping (chapter 18).
• Secured: least-privilege identity, no long-lived credentials, secrets from a manager (chapters 6 and 7).
• Affordable: an estimate of its monthly cost, and tags so it appears on the bill (chapter 19).

Change, without ceremony

Heavyweight change-approval boards mostly add delay: they batch changes, which makes each one riskier, and they ask people with less context to approve work they can't evaluate. The alternative that actually reduces risk is the one this volume has been building: small changes, automated checks, staged rollout, fast rollback, and a record of what changed and when, annotated on the same dashboards you watch during an incident. Where you do want a brake, tie it to the error budget rather than the calendar: freezes triggered by a number are negotiated once, not every time (chapter 15).

Practise before you need to

Schedule the drills you know you need: a restore (chapter 18), a failover, a rollback, an on-call handover, and a game day where you break something in a controlled way and run the incident process for real. Rotate who leads them, and use them to onboard: shadowing a drill teaches more in an afternoon than a month of documents. Write down what didn't work, because the drill's findings are the point, not the successful recovery.

If you have just inherited a system

A closing order of operations, for the common case where all of this needs to exist and doesn't yet:

1. Find out what breaks and who is woken by it. Read the last three months of incidents and pages.
2. Make one dashboard that answers whether the product is working, and one alert that fires when it isn't.
3. Make deploys reversible. Everything else is easier once you can undo.
4. Time a restore. Whatever the result, you will know something important.
5. Write down the three things you did last month by hand, and automate the most frequent one.
6. Set one SLO on the most important journey, and start showing the budget to whoever prioritises the work.
7. Then containers, clusters, pipelines and everything else in this volume, in whatever order the pain dictates.