Start with the SLO. The SLO is the service-level objective: the reliability the system is committed to deliver. The SLO is the input to the error budget, the SLO is the input to the alert, the SLO is the input to the capacity planning, the SLO is the input to the incident response. The SLO is the document the engineering team and the business team agree on; the SLO is reviewed when the reliability changes.
Resilience Engineering
The system is something that will fail. The observability is the evidence of how it is failing. The chaos engineering is the practice of testing the failure mode. The incident response is the discipline of recovering gracefully. The post-mortem is the discipline of learning.
On this page
§ 01 ·
What resilience engineering is
Resilience engineering is the discipline of designing, building, and operating systems such that the system continues to deliver value in the presence of failure. The discipline covers the failure model (the explicit statement of what can fail, how it can fail, and what the system does when it fails), the observability (the evidence the system produces about its own state), the chaos engineering (the practice of testing the failure mode before the failure mode tests the system), and the incident response (the discipline of recovering gracefully when the failure mode wins).
The most important property of a resilience engineering practice is that the system is treated as something that will fail. A system designed with the assumption that the components are reliable is a system that has not been designed for the reality that the components are not reliable. The right answer is to design for the failure: the redundancy, the circuit breakers, the bulkheads, the timeouts, the retries, the backpressure, the graceful degradation.
The two failure modes that come up most often are: (1) the system is designed to be reliable, and the failure mode is not designed, and the failure produces a cascade that the system could have contained; (2) the system is designed to be resilient, but the failure modes are not tested, and the resilience is theoretical, not empirical. The right answer is to design for the failure and to test the design before the failure tests the design.
§ 02 ·
The architecture
The architectural primitives that have held up:
The bulkhead. A failure in one component is contained to that component. The bulkhead is the pattern (separate thread pools, separate queues, separate rate limiters, separate data stores) that prevents the failure from propagating. The bulkhead is a property of the architecture, not a property of the configuration.
The circuit breaker. A call to a failing component is short-circuited when the failure rate exceeds a threshold. The circuit breaker is the pattern that prevents the system from overwhelming the failing component with retries. The circuit breaker has three states (closed, open, half-open) and a documented recovery time.
The timeout. A call to a slow component is timed out. The timeout is the pattern that prevents the system from being held hostage by the slow component. The timeout is shorter than the request budget and longer than the typical response time.
The retry with backoff. A call to a transiently failing component is retried. The retry has a backoff (exponential, with jitter) and a maximum number of attempts. The retry is the pattern that handles transient failures (network glitches, brief resource contention) without overwhelming the component.
The backpressure. A slow consumer signals backpressure to the producer. The backpressure is the pattern that prevents the producer from overwhelming the consumer. The backpressure is a property of the protocol (the consumer acknowledges what it can accept, the producer respects the limit).
The graceful degradation. A failure in a non-critical component is handled by degrading gracefully, not by failing the request. The degradation is the pattern that allows the system to continue to deliver value when a component is unavailable. The degradation is documented (what gets degraded, what stays available).
§ 03 ·
State of the practice
3
SRE book chapters that have changed how the industry thinks about reliability (Chapter 1: principles, Chapter 5: eliminating toil, Chapter 27: the post-mortem)
Beyer, Jones, Petoff, Murphy, 'Site Reliability Engineering', O'Reilly, 2016
5
Common resilience patterns (bulkhead, circuit breaker, timeout, retry with backoff, backpressure) — each addresses a specific failure mode
Industry observation; from 'Release It!' (Michael Nygard)
1
Right answer to 'how much resilience' — the amount that the business needs, documented in the SLO
§ 04 ·
Questions we get asked
The amount that produces evidence the team can act on, without producing outages the team cannot afford. Start in a non-production environment (a staging environment, a chaos engineering lab). When the chaos engineering has been validated in non-production, promote it to production (in a controlled scope, at a controlled time, with a controlled blast radius). The chaos engineering is a practice, not a project; the practice is continuous, the experiment is documented, the finding is fixed or accepted.
Blameless, focused on the system, and forward-looking. The post-mortem is the discipline of learning from incidents, not the discipline of assigning blame. The post-mortem asks: what happened, why did it happen, what was the impact, what did we do to recover, what do we change to prevent the recurrence. The post-mortem is shared (the team learns, the organisation learns); the post-mortem is tracked (the preventive action is owned, the preventive action is implemented, the preventive action is verified).
The one that gives the team the evidence it needs to debug the system. The three pillars (metrics, logs, traces) are the canonical framing. The metrics are the aggregate view; the logs are the event view; the traces are the request view. The right answer is to start with the metrics (the RED method: Rate, Errors, Duration, for services, the USE method: Utilisation, Saturation, Errors, for resources) and to add logs and traces as the debugging requires. The stack is a property of the paved road; the team uses the paved road by default.
As a practice, not a punishment. The on-call rotation is fair (the rotation is shared, the rotation is documented); the on-call is sustainable (the on-call is one person, with a backup, with a clear escalation path); the on-call is reviewed (the alerts are tuned, the alerts are not too noisy, the alerts are not too quiet); the on-call is not the only source of reliability work. A team that is always on-call is a team that is burning out; a team that is never on-call is a team that is not learning.
The one that the business and the engineering team agree on. The error budget is the inverse of the SLO (99.9% SLO = 0.1% error budget = 43 minutes per month). The error budget is the input to the alert (the alert fires when the error budget is exhausted), the input to the capacity planning (the capacity is planned to fit within the error budget), the input to the change velocity (the change velocity is throttled when the error budget is exhausted). The error budget is the discipline that prevents the engineering team from trading reliability for velocity, and prevents the business from trading velocity for reliability.
§ 05 ·
Patterns we design our practice to avoid
A system designed with the assumption that the components are reliable is a system that has not been designed for the reality that the components are not reliable. The right answer is to design for the failure: the redundancy, the circuit breakers, the bulkheads, the timeouts, the retries, the backpressure, the graceful degradation.
A chaos engineering practice that is run once is a practice that has produced a finding list, not a resilience practice. The chaos engineering is continuous: the experiments are run regularly, the experiments are run in production (in a controlled scope), the findings are tracked. The chaos engineering is a property of the operating model, not a property of a project.
A post-mortem that assigns blame is a post-mortem that produces cover-your-ass behaviour, not learning. The right answer is the blameless post-mortem: the focus is on the system, the focus is on the contributing factors, the focus is on the preventive actions. The person who made the change is not the cause of the incident; the system that allowed the change to be made without safeguards is the cause of the incident.
A team that has on-call but no SLO is a team that is reactive, not proactive. The SLO is the commitment; the error budget is the discipline; the chaos engineering is the test; the post-mortem is the learning. The on-call is the operational practice, not the reliability practice.
An observability stack that is tightly coupled to a single vendor is an observability stack that is hard to migrate. The right answer is to standardise on open formats (OpenTelemetry for traces, PromQL for metrics, structured logs) and to choose the vendor that supports the open formats. The vendor can change; the practice cannot.
An alert that fires on every transient condition is an alert that produces alert fatigue. An alert that fires only on the customer-impacting condition is an alert that produces attention. The alert is tuned to the SLO; the alert is documented; the alert is reviewed. The alert is not the on-call; the on-call is the response to the alert.
§ 06 ·
Evidence & references
Public frameworks and writing that inform our practice.
Beyer, Jones, Petoff, Murphy, O'Reilly, 2016
The book that established the SRE discipline. The treatment of the SLO, the error budget, the post-mortem, the toil, the on-call is the canonical reference. The book is the foundation; the practice is the implementation.
Release It!
Michael Nygard, Pragmatic Bookshelf, 2018 (2nd ed.)
The book that established the resilience patterns. The treatment of the bulkhead, the circuit breaker, the timeout, the retry, the backpressure is the most rigorous available. The book's contribution is the framing of stability as a property of the architecture, not a property of the configuration.
Basiri, Behnam, de Ruiter, et al., O'Reilly, 2021 (open access)
The book that established the chaos engineering discipline. The treatment of the experiment design, the blast radius, the steady-state hypothesis, the learning is the canonical reference.
Observability Engineering
Majors, Fong-Jones, Miranda, O'Reilly, 2022
The book that established the observability discipline. The treatment of the three pillars (metrics, logs, traces), the structured events, the high-cardinality data is the most useful recent reference. The book is the bridge between the SRE book and the vendor literature.
PostgreSQL Global Development Group, current
The canonical reference for write-ahead logging, the durability primitive. The pattern (every change is written to a log before it is applied; the log is the source of truth) is the foundation of every database's reliability story. The reference is dense; the principle is universal.
Building a resilience practice?
If you are weighing a resilience engineering initiative, evaluating a chaos engineering tool, or trying to make an existing on-call practice more than a reactive service desk, we are useful at the boundary between the failure model and the implementation. A short conversation is the right next step.
Learn more