SRE book notes: Eliminating Toil

These are the notes from Chapter 5: Eliminating Toil from the book Site Reliability Engineering, How Google Runs Production Systems.

This is a post of a series. The previous post can be seen here:

SRE book notes: Service Level Objectives

Hercules Lemke Merscher Jan 12 3 min read

#sre #books #slo

So what is toil? Toil is the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows. Not every task deemed toil has all these attributes, but the more closely work matches one or more of the following descriptions, the more likely it is to be toil

We dont need to go that far to find toil in our daily work. A very simple example of a task that can be considered toil is a (semi)manual step-by-step deployment process.

Who never had to have installed a set of tools in the laptop to build and deploy a program at some point in time, and perform some actions in a predefined order?!

Its the kind of thing that becomes more annoying the more the company grows, and nobody is willing to do it.

At least 50% of each SREs time should be spent on engineering project work that will either reduce future toil or add service features.

We share this 50% goal because toil tends to expand if left unchecked and can quickly fill 100% of everyones time.

Toil isnt always and invariably bad Its fine in small doses, and if youre happy with those small doses, toil is not a problem. Toil becomes toxic when experienced in large quantities. If youre burdened with too much toil, you should be very concerned and complain loudly.

Your career progress will slow down or grind to a halt if you spend too little time on projects. Google rewards grungy work when its inevitable and has a big positive impact, but you cant make a career out of grunge.

Be attentive if youre the kind of professional who starts to happily accept too much toil to help your colleagues. People around you will be super happy that youre dealing with the boring work for them, but eventually, you could be setting a trap for yourself, as without noticing, they will let you deal with the toil all the time to the point you will not have time to work on projects that will make you progress in your career.

People have different limits for how much toil they can tolerate, but everyone has a limit. Too much toil leads to burnout, boredom, and discontent.

Excessive toil makes a team less productive. A products feature velocity will slow if the SRE team is too busy with manual work and firefighting to roll out new features promptly.

Even if youre not personally unhappy with toil, your current or future teammates might like it much less. If you build too much toil into your teams procedures, you motivate the teams best engineers to start looking elsewhere for a more rewarding job.

New hires or transfers who joined SRE with the promise of project work will feel cheated, which is bad for morale.

Most probably, the best engineers will dodge the bullet before even joining the team by asking the right questions during the interview process - considering youre being frank with them.

Few engineers are motivated to jump into a big pile of toil and accept the challenge of dealing with and reducing it to acceptable levels.

This chapter resonated a lot with some experiences I had in the past. I wish I had read this chapter before. At least now I have a piece of literature to recommend.

So, better keep toil under acceptable thresholds!

If you liked this post, consider subscribing to my newsletter Bit Maybe Wise.

You can also follow me on Twitter and Mastodon.

Photo by Tim Gouw on Unsplash