Site reliability engineering (SRE) is a discipline in which automated software systems are built to manage the development operations (DevOps) of a product or service. In other words, SRE automates the functions of an operations team via software systems.

The main purpose of SRE is to encourage the deployment and proper maintenance of large-scale systems. In particular, site reliability engineers are responsible for ensuring that a given system’s behavior consistently meets business requirements for performance and availability.

Furthermore, whereas traditional operations teams and development teams often have opposing incentives, site reliability engineers are able to align incentives so that both feature development and reliability are promoted simultaneously.

Basic SRE principles

In this article, we’ll cover key principles that underlie SRE, provide some examples of those key principles, and include relevant details and illustrations to clarify these examples.

Embrace risk

No system can be expected to have perfect performance. It’s important to create reasonable expectations about system performance for both internal stakeholders and external users.

Key metrics

For services that are directly user-facing, such as static websites and streaming, two common and important ways to measure performance are time availability and aggregate availability.

This article provides an example of calculating time availability for a service.

For other services, additional factors are important, including speed (latency), accuracy (correctness), and volume (throughput).

An example calculation for latency is as follows:

• Suppose 10 different users serve up identical HTTP requests to your website, and they are all served properly.

• The return times are monitored and recorded as follows: 1 ms, 3 ms, 3 ms, 4 ms, 1 ms, 1 ms, 1 ms, 5 ms, 3 ms, and 2 ms.

• The average response time, or latency, is 24 ms / 10 returns = 2.4 ms.

Choosing key metrics makes explicit how the performance of a service is evaluated, and therefore what factors pose a risk to service health. In the above example, identifying latency as a key metric indicates average return time as an essential property of the service. Thus, a risk to the reliability of the service is “slowness” or low latency.

Define failure

In addition to measuring risks, it’s important to clearly define which risks the system can tolerate without compromising quality and which risks must be addressed to ensure quality.

This article provides an example of two types of measurements that address failure: mean time to failure (MTTF) and mean time between failures (MTBF).

The most robust way to define failures is to set SLOs, monitor your services for violations in SLOs, and create alerts and processes for fixing violations. These are discussed in the following sections.

Error budgets

The development of new production features always introduces new potential risks and failures; aiming for a 100% risk-free service is unrealistic. The way to align the competing incentives of pushing development and maintaining reliability is through error budgets.

An error budget provides a clear metric that allows a certain proportion of failure from new releases in a given planning cycle. If the number or length of failures exceeds the error budget, no new releases may occur until a new planning period begins.

The following is an example error budget.

Planning cycle	Quarter
Total possible availability	2,190 hours
SLO	99.9% time availability
Error budget	0.1% time availability = 21.9 hours

Suppose the development team plans to release 10 new features during the quarter, and the following occurs:

• The first feature doesn’t cause any downtime.

• The second feature causes downtime of 10 hours until fixed.

• The third and fourth features each cause downtime of 6 hours until fixed.

• At this point, the error budget for the quarter has been exceeded (10 + 6 + 6 = 22 > 21.9), so the fifth feature cannot be released.

In this way, the error budget has ensured an acceptable feature release velocity while not compromising reliability or degrading user experience.

Set service level objectives (SLOs)

The best way to set performance expectations is to set specific targets for different system risks. These targets are called service level objectives, or SLOs. The following table lists examples of SLOs based on different risk measurements.

Time availability	Website running 99% of the time
Aggregate availability	99% of user requests processed
Latency	1 ms average response rate per request
Throughput	10,000 requests handled every second
Correctness	99% of database reads accurate

Depending on the service, some SLOs may be more complicated than just a single number. For example, a database may exhibit 99.9% correctness on reads but have the 0.1% of errors it incurs always be related to the most recent data. If a customer relies heavily on data recorded in the past 24 hours, then the service is not reliable. In this case, it makes sense to create a tiered SLO based on the customer’s needs. Here is an example:

Level 1 (records within the last 24 hours)	99.99% read accuracy
Level 2 (records within the last 7 days)	99.9% read accuracy
Level 3 (records within the last 30 days)	99% read accuracy
Level 4 (records within the last 6 months)	95% read accuracy

Costs of improvement

One of the main purposes of establishing SLOs is to track how reliability affects revenue. Revisiting the sample error budget from the section above, suppose there is a projected service revenue of $500,000 for the quarter. This can be used to translate the SLO and error budget into real dollars. Thus, SLOs are also a way to measure objectives that are indirectly related to system performance.

Using SLOs to track indirect metrics, such as revenue, allows one to assess the cost for improving a service. In this case, spending $10,000 on improving the SLO from 95% to 99% is a worthwhile business decision. On the other hand, spending $10,000 on improving the SLO from 99% to 99.9% is not.

quent changes using version control and code reviews.