The Real Cause Of Your Workplace Headaches

I am going to take a wild guess that you recently had an argument or heated debate with a colleague. I’m sure the topic was riveting. You both made logical points. You both felt strongly about your stance. You continued to argue and then both sides felt unheard. What you probably didn’t realize is that the root of your disagreement actually stems from how you each interpret and manage risk. This problem comes from the fact that the brain isn’t exactly rational; Actually, it’s not rational at all. If you don’t believe me, go pick up Dan Ariely’s book Predictably Irrational. Fear of pain leads us to avoid certain circumstances due to our past experiences. And these past experiences dictate our reality. Our fear of risk served more of a purpose when we were fighting off lions for survival. This fear serves less of purpose when we’re fighting off Michael from marketing for that last cup of coffee. As evolved as we are as humans, we are in many ways still driven by basic human instinct.

You may not be aware but you’re essentially weighing risk versus reward all day, every day. Even if risk no longer means being eaten to death by a lion. The issue is, your brain can struggle with perceived risk vs real, life-threatening risk.

• “Should we add that cool commenting feature?”
• “Should we deploy on a Friday?”
• “Should we promote John to VP?”
• “Should we fire half that team?”

These examples are wide-ranging but are all related to risk. The counter-argument to the above proposals sound similar to the following:

• “We will be wasting developer resources because it is unlikely that the new commenting feature will bring in new users”
• “If we deploy on Friday, there will be outages all through the weekend with no one around”
• “John makes people upset, employees will leave under his leadership”
• “We can save money and the team will probably be able to function”

If you peel back the onion for most arguments it all has to do with risk and its potential for harm. And yet we take little time to understand and even less time to measure it. Yes, this might mean you will actually have to put in some extra work. Work that many of us don’t want to do. To make your life easier, I recommend you take that extra step, don’t be intellectually lazy here. It’s too damn important. In computer systems, it’s rather easy to measure risk as we can use the colossal amounts of machine data emitted from our systems. As we move into people or social-based systems, measuring risk becomes harder to achieve since we have to predict much further into the future how our policies will affect a group of people. But before we can measure risk, we must first define risk. Let’s define it as the following:

Risk = failure probability x damage related to the failure

We recommend sitting down with your team and defining risk for yourselves. We also suggest thinking about ways to reduce damage related to the failure rather than eradicating failure. There are many frameworks out there to help you define and limit risk such as the error budget framework from Google’s SRE handbook. The next step would be in figuring out people’s tolerance for risk. This is the key, where we all differ.

Most engineers in the “risk reduction” game focus on reducing the failure probability to 0 such that there is no risk in the system. This however has unintended consequences. They rarely discuss and strategize about how to reduce damage related to the failure and how to calculate it. In many companies, they refer to this methodology as experimentation. But in most organizations, experimentation stops with product and features, not software development methodologies and certainly not finance or marketing. This is an important point, on the path to seeking a failure rate of 0% you will inevitably be creating collateral damage elsewhere. Every decision has a cost.

It’s also important to note that there are different types of failure. What we refer to is safe to try or not safe to try. Will this failure lead to irrevocable damage? If the answer is yes, then we do our best to get as close to zero. Say, when human lives are on the line or the company will collapse. If the answer is no, it’s important to get clear on the type of risk and the tolerance for it. Engineers tend to lump all failure into a giant bucket of risk which results in pointless debates and sluggish progress.

If we can’t accurately quantify or approximate risk, we’re essentially speaking different languages. How you think about risk in a system may be entirely different from how I consider risk. For example, what are the proper risk considerations given on an e-commerce website? Is it error rates to the notification system? Is it a broken checkout flow? This is the root of most arguments between Dev and Ops. Does this change introduce a real risk to the system and if so, how much? Given that nobody can prove, with data, how much risk the new change will introduce we would rather not accept the risk at all.

The Slipperiest of Slopes

Without talking about and measuring risk we tend to extrapolate to the worst-case scenario. Our brains quickly paint a “doom & gloom” scenario because your brain is trying to protect you from harm. Thus your brain will leverage your confirmation bias until it achieves its intended outcome: complete and total safety.

• “If we let development team A deploy whenever they want, then we have to let all teams deploy whenever they want.”
• “Because development team A had a security hole, all teams must now go through a stringent security policy regardless of what the application does.”

There are countless arguments fought about risk and it’s reduction. By not properly naming and quantifying risk it opens the doors to a multitude of arguments that lead to nowhere. It also leads to extremist conversations and binary outcome thinking.

Understanding Your Team’s Risk Tolerance

When making decisions that are safe to try, come up with a shorthand with your team. First: define the level of risk in this decision and secondly people’s comfort with risk in this given context. Context is king, people will always have different feelings about different situations. You can use a 5 point system just with your hand, ask your team “how much risk do you want to take here?” 5 being the highest amount and 1 being the lowest. Have them hold up 1–5 fingers. If anyone has up a 1 or 5, I would ask them to share their thoughts. What are they worried about? What might the team be missing? You can figure out with your team what works here. But the point is to get clear on the foundations of an issue before trying to resolve it. By taking this extra step, you will not only save yourself a lot of headaches but you will also increase the level of trust and communication with your coworkers. As you get an understanding of where the tension lies (comfort around this specific risk), the next step is to find a way to measure that risk. Turns out the risk of misunderstanding risk is really fucking high. Measuring it will help everyone get on the same page. Oh, and since you were wondering, I told Michael from marketing that I thought something was wrong with the coffee and he gladly walked away. I won.