Join Murray Robinson and Shane Gibson in a conversation with Adam Thomas about Product Survival Metrics. Empower the product team to make decisions quickly in uncertain situations by negotiating clear boundaries for the product team. Negotiate boundaries with stakeholders early. Use data to track your progress. When your heading towards a boundary make decisions to stop, pivot or invest. Product development is not a linear path. Agile product development is not a methodology that you buy it’s a way of working that you find. Use decision journals to document and review decisions.

Read along you will

Shane: Welcome to the No Nonsense Agile Podcast. I’m Shane Gibson. 

Murray: And I’m Murray Robinson. 

Adam: I’m Adam Thomas. 

Murray: Hi Adam. Thanks for coming on. We wanna talk to you about product survival metrics today, but why don’t you kick us off by telling us a bit about your background and experience. 

Adam: Sure. So I started off as a, content creator writer, for a website called All hiphop.com. I ended up creating a website called the gamer studio.com, which I built from scratch using php, and next thing you know, it was a startup. From there ended up in product management and afterwards took me to a whole bunch of places over the course of about 15 years to where we are today, where I am the founder and principle of a consulting firm I call Approaching One, where I help companies do three things and try to get them to do them well.

One is hire product people, next is help companies launch stuff, and the third thing that I help companies do is learn how to not stick on bad decisions. I’ve created this methodology called Survival Metrics that helps people build the muscles inside of companies to be able to make changes when they matter instead of making it all at once.

Murray: Well, What are product survival metrics then? 

Adam: So it’s a methodology to help you avoid sunk cost bias. It focuses on three pillars , which is decision velocity or speed; data literacy or being data informed, and political safety. And whether someone should Stop what they’re doing completely pivot which is change direction or invest, meaning that we should stop other things in order to invest deeply into this. With the default state of course being continue, but teams don’t tend to have a problem with continuing. It’s everything but continuing where the issues come into play.

Murray: Can you be specific and give us some examples? 

Adam: Sure. So one of the first companies I worked with had a huge windfall in first six years of their existence. It was bootstrap, but they had 6 million of profit. They had pretty high NPS scores about 45. And they had high retention. They got really antsy and said, Hey, there’s some big projects that we wanna do. And so let’s let’s just pick a few and go with it Okay, let’s redo our infrastructure and do this AI project. However, about two years later after they had scaled to take on those projects, they ended up with massive layoffs. They had to cut about 50% of their staff and as a result, they got rid of their entire product team. I came in with a mandate to change culture. We just spent two years investing all this energy into this thing, this thing failed. And for some reason during that entire time, no one put their hand up and said, This isn’t really serving us. 

And so that became the first pillar of survival metrics, which is political safety. Political safety is not just the conversations between teams. It’s an understanding of the incentives that drive them. From there I got an understanding of where the team started to fumble. Teams weren’t talking. There is no feedback loop. 

The second thing is they had a lot of gut-based decisions that were driving the conversation. They didn’t really have a way of tracking or understanding or talking about the data that they had, nor were they using that data in decisions. And in fact, that team is pretty ignorant to how teams measure data or how they understand what data is important.

So this is where something called a decision journal comes into play. A decision journal is a thinking tool that lists out what a decision is, you list out what data was used in order to make that decision, and you write out the context and how that decision was made. This is important because the human brain gamifies everything. It’s just how we’re wired, and if you do not have a place where your major decisions are laid out you’ll never have a chance to really iterate and get better.

Those decision journals led to us understanding the incentives. And as you start to develop that data literacy, you start to find out what the metrics are, and that becomes your base of the survival metrics. And then lastly, you gotta act on these things, and that’s where decision speed comes into play. Most companies fail because they don’t act fast enough and that speed costs money.

In this company’s case because they didn’t have an understanding of incentives, nor did they have data journals. They didn’t have spaces in which to actually discuss the next step? Where are we going next? And how is this important to the actual goals that we’re setting? Without these kind of check-ins and feedback loops the system settles and you end up creating things that aren’t data driven. Your data utilization rates are the inverse of how many decisions you’ve stopped in the last six months.

Murray: So the three categories of things you look at are political safety, data literacy, and decision speed. Is that right? 

Adam: yes. 

Murray: Okay. But you’ve called this system survival Metrics. A metric to me is a measurement, and I haven’t heard you talk about measurements yet.

Adam: To get good metrics, you need to create good earth or good foundation for which metrics can last or work. 

Shane: So one of the problems we have in the agile world is the idea of estimation, velocity work have been done. And so one of the, anti patterns we see is as soon as people start estimating they get held to account, we don’t worry about anything else other than the rate they’re doing the work. What I think I’m hearing is the same pattern from you that if we focus purely on the measurement, on the metric, and we don’t worry about everything that surrounds it, then we’re not gonna succeed our outcome of making the right decisions.

So we need to focus on those three pillars of change. If we can’t change our decisions at speed, if we can’t change ’em based on data and if we don’t have the safety to make them, then it doesn’t matter if you’ve got a North Star number who gives a shit because you’re not gonna be able to move the boat because there’s no cultural ecosystem that allows you to move the boat. And then it gets worse because like velocity and estimation, that North star metric, that becomes the holy grail. That’s the gospel. That’s the thing. You can never say it’s wrong because actually it’s the only shining light in the organization. So I think that’s what you’re saying is that it 

Adam: Nailed it Shane, this comes from experience playing burn down theater. Perfect burn down charts, but what have we produced? What has mattered? What outcomes have we driven? 

Murray: Yeah, so political safety and data literacy are required so that you can make good decisions quickly. How do you set your goals and your measures for political safety? 

Adam: The home for political safety is in the vision statement or public documents. How do we present ourselves outward to the world? Cuz that’s our idealised state. There’s a conversation you can have with a leader to talk about who we are. If you go talk to enough CEOs, they’ll tell you that no one listens to them. They don’t see the big rocks being followed in the things that they see go out the door, especially when they’re farther and further away from a product being produced. This is how you end up with poop and swoop moves from leadership. 

Looking at those documents, you can start to figure out what are the red lines in a company. And so you can start to actually put them to the test. You can create a metric that is specifically tailored to that value statement and put that in front of teams and in front of everyone.

Let me give an example. Let’s say there’s a company that has a real problem with third party data, they talk about it all the time. We are a private company. We don’t sell our data. We don’t buy other people’s data. This is really important to us. As this company grows, that message is gonna get watered down, even if the founder may still believe this, in fact, this might be something he’s telling the board. This might be something he’s telling venture capitalists, to raise more money. We’re private company. Meanwhile, there are companies, on the line that are creating product and software that may use that data. 

If you have a metric that we stop immediately when third party data enters the equation, that makes that really clear. The ceo seeing that work makes them feel like they’re being listened to, this is the company that I built or this is the company that I run. This is the company that’s important to me. If it’s messy, they may have this gnawing feeling underneath where they go, I don’t really feel like we’re living our values and I can’t really see it. No one talks about them and so I gotta start meddling.

Murray: Yeah. So you said the rule is that as soon as we see third party data, we stop what we are doing. Who sets that role? 

Adam: So in an ideal world, everyone would be responsible. However, in real life, in practice, this is not really the case. It’s usually an engineering manager or a product person that would set the tone. Stop immediately means we’re not doing anything around this. If this is the only path forward, we stop the project with the understanding that this is a thing that makes us stop. Now you don’t have to do any other kind of dance of what the next action is, which is where a lot of teams end up fumbling.

Shane: Let me play this one back again. So I like this idea of stop, pivot, invest, and continue. So if we don’t do anything, everybody’s gonna continue, stop means we’re stopping we’ve hit the wall you’re not going any further. Pivot means, we’re gonna u-turn and invest means we stop something else. And then you’ve just linked it to values of the organiz. The core foundational theories, hypothesis, beliefs of the founders or leaders of the organization. We could then use metrics to codify those values and those hypotheses, and then use them to drive one of those four actions.

I’ll give you an example. In our startup, we do not want to hold personally identifiable information. We don’t wanna do it because it’s dangerous. We have a bunch of technology that if we do have it, we can mask it, to protect it it’s automatically encrypted. Some things we do by default and some things we would add if we are holding any of those high risk pieces of information. So that’s philosophy, our belief, no PII data unless we agree it with our customer. And then we put a whole lot of belts and braces around those particular values. So that could be a metric, as soon as that data turns up from a customer, that metric triggers a stop for that customer and we go have a conversation with them.

Now, we may change our philosophy over time, cuz we’re bootstrapping and still finding product market fit. We might find that actually the problem everybody’s got is nobody will hold their PII data. So we are then going to pivot and invest in that. Is that the way you think of it? 

Adam: Absolutely, it’s a way to keep ourselves honest over time. Any framework or methodology, or information we add to a system, should move a needle around a decision that a team needs to make. 

Shane: So the survival metrics, this idea of documenting things we wanna measure so we can decide whether we’re gonna stop, pivot, invest, or continue. They’re forcing functions, so at the moment we don’t tend to force it. We tend to treat it as posters on the wall with our values or a strategy slide about what we’re gonna do to change the world. We don’t make them actionable, they’re fluff. So by tying these conscious metrics to it we can force action when we need to force it. And then in theory, over time, the culture of the organization will start to behave that way. 

Like, the Amazon memo, one pager where they’re documenting the reason that they think some action should be taken with as much data as possible versus, a PowerPoint with a couple words and somebody spending 45 minutes talking through with fluff about why it’s a great idea. 

Adam: That is correct. No more long presentations. We’re gonna sit here with a document that whoever calls a meeting has to write. We don’t have it until you write it. And then we sit down and we read it together, and then the rest of the time is really about that decision that we have to make. That’s the genius of the Amazon memo. Much like survival metrics or much like retrospectives are another way of this happening. And then we come out with a better idea than where we started.

Shane: So if I use another analogy for a survival metric, it’s sounding to me like a sensor. So if you think about a factory and it’s automated, we have a bunch of sensors, something happens on the conveyor bill to the item, it gets out of alignment, sensor goes off, convey belt stops. We’ve got a car, it’s autonomous, it’s driving down the road, it’s got a bunch of sensors. One of the sensors is triggered, the car stops, it changes its behavior. So, how do we think about the sensors that we can use and the way we’re working, and if they’re triggered, we want action to be taken. The action being stop, pivot, invest, or continue. The trick is thinking about that early, with data. Okay, what am I worried about? What would make me change my mind? Okay, let’s write that up as a sensor, as a metric, then how do we get data to figure out whether that sensor’s been triggered or not? And then once it’s triggered, how do we make sure the team, make their change. 

Adam: That’s perfect. The idea is to sense and adjust.

Murray: Can you tell us more about what you are looking at around decision spade? 

Adam: Yes. So decision speed, I think there are three major things around decision speed that hold up teams. And the first one is telling people that there’s a decision to be made. Without prompting, people won’t realize what decision you’re asking to be made. Without talking about what decision needs to be made, then you’re losing decision velocity.

The second thing that people tend to do is they do not lay out the options in front of them or what options cannot be used. If things are not on the table, then it’s very important that you lay them out on the table. That takes some work by the person that’s asking the group for the decision. However, this speeds up the decision velocity way more than sitting and creating a PowerPoint that’s talking about nothing. It’s really important that those things are well defined, the reason why you stop, pivot and invest is there, there’s no guessing about what those words mean, it’s very clear. And so by laying those things out, you’ll see an increase in decision velocity for a team. 

The other thing around decision velocity is risk identification. Many of the decisions that we make are way simpler than we give it credit for. There’s a smaller portion of them that have significant risk and need to be clarified. We should probably break those problems down a bit more.

Shane: Okay. So one of the things we know is when people want to prioritize the work to be done, they tend to pretend to be data driven. So we’ll come up with some metrics. We’ll go, Oh, let’s look at value to the organization of doing that work. And let’s look at risk of the work being done and let’s look at how long it’s gonna take the team to do it. And, we’ll put a couple of columns in a spreadsheet and we’ll put some numbers in there and we’ll come out with a ranking. And then we know that as soon as we talk to the stakeholders the hippo, highest paid opinion in the room will go. Now it’s that one. So I can see that there’s a danger with these metrics of that behavior happening.

But if we don’t do that, the other option is we just say to somebody, what’s the most important? What’s your gut say? And we run with that. If we can get the work done early enough and test the feedback, it’s actually probably less time intensive and better value than trying to score these things and ignoring the scores. So how do you deal with that? How do you deal with this idea that we sometimes just gamify the metrics.

Adam: I think it’s the role of somebody with the ability to be very clear about those things, whether that’s a product leader or a cto. if this is something that has to be done, why are we pretending? That’s really a sign of poor leadership. If you’re working in an environment where this is just not gonna fly, I’m gonna guess there are other things that are massively wrong and your stomach hurts every day when you walk in, and so you should probably leave. 

Douglas Hubbard talks about how to measure anything. And it talks about what a good metric is. And I think it’s up to someone to be the keeper of this. And the four things that are really important Is it clear what’s happening now? What’s the context? Is it clear what the value of additional information is? What’s the value of this metric? How does it link back to our strategy and mission? And then If this happens, then that. If you have the scale it’s a good time to take it to somebody else that’s a neutral party and say, What do you think? 

It’s hard to call yourself on gamifying, this is where I think writing things down is helpful, like back to the decision journal and saying this is the important parts, these are the hard parts. And just being frank there is useful. But going and talking to a mentor or a third party that’s neutral an old boss, even like a group, for example I am one of the facilitators for Product Tank here in nyc. Finding a few folks from your local group event and then talking about that can help you get some understanding and some clarity.

Murray: Okay. Summary is Shane.

Shane: Okay, So we are looking to empower the team to make decisions about stop doing what they’re doing, pivoting to something else or investing in something else. And by investing me mean stop investing in one thing, to use that money or that time to invest in another thing.

If we don’t do one of those three actions, our default action will be to continue. To do that we need the ability or the culture in the organization or the team to be able to make change at speed for that change to be informed by data and for them to have the political safety to be able to make that decision and not get fired. So that empowerment of the team that we were always looking for. Why would we wanna do that? Because if we don’t, we’re gonna fail. And product’s gonna take the sword. 

The other thing you talked about, was that philosophy of observe then act. Often people go in with an answer, you talked about the way that you typically go in and watch what’s happening first. And then the fact that some of these techniques are forcing functions like we have with the retros, with the standups. So it’s a way of saying we can define these survival metrics and if they get triggered, then we need to do a stop pivot or invest decision because continuing on is not a safe path. If we can’t see things are failing from those metrics, we’re probably not gonna go fix it cuz we think everything is happy and it’s not.

I like the idea that product development’s not a linear path. We’re not going from A to B. It’s always winding, it’s always changing. Same with agile. Agile’s not a methodology that you buy on an a three piece of paper implement and do 90 day bloody increments. It’s a way of working that the team need to find. And there are a bunch of patterns and things they can use that have value. 

I love the idea of decision journals . So the idea that we are gonna document what data we used to make a decision and what the context was when we made that decision to, hold ourselves accountable that we did or did not use data to make the decision. Also, that’s probably a good place for us to state the immutable decisions, the ones where we go, Actually, I’m not gonna empower the team, this is a boundary, this is a stop and it might change later, but I’m gonna be very clear when it does change. So I like that kind of lens of the decision journal and that muscle memory of just writing these things down to yourself, even if you don’t share it with other people, is probably a good practice to start with.

I think the term survival metrics makes it confusing. And as soon as we hear metrics, we probably jump straight into, KPIs, not even OKRs, and I think we’re not talking about just a metric, we’re talking about a way of working and a bunch of patterns that you can use. But if I come back to that idea of a sensor, how can we put data driven sensors into what we are doing? And they can be driven from some product decisions we’ve made, some addressable market decisions, we’ve made some cultural decisions we’ve made. If we can codify those where the data tells our teams we’ve got a problem and they need to make a decision of stop, pivot, or invest because they’re not allowed to just continue, then there’s some value in that pattern. Murray, what about you?

Murray: Yeah, I think there’s no doubt that product managers need to make good decisions quickly under situations where there’s a lot of uncertainty, particularly in startups. And in order to do that, you need to understand what data you have available and work out what’s the right data to use and you need to be aligned with the objectives of the organization. So you need to have established what those objectives are with your stakeholders. That all makes sense to me. What I’ve been struggling with is the term survival metrics, cuz I don’t see any of these things as really being metrics. They are categories to think about and some things under each category that might help, like decision journals. So I kept asking about metrics and I didn’t really get any metrics because it’s not really about metrics.

Shane: I don’t think I agree with you, Murray. I think we’ve got a bunch of lenses we use to decide what metrics are important, which ones are our true survival ones. I don’t think it’s come through very clearly, but I think that’s what it is, so if I go back to that example of pii, If I set that as a core survival metric for our startup, it’s immutable, it’s one that when that data turns up, that metric is triggered. But it’s not based on vanity metrics. I don’t look at things that make me feel good because the number’s going up. Because if the number’s going up, we’re gonna continue. I don’t need to stop, pivot, or invest.

Murray: Yeah, but we haven’t talked about metrics at all really. 

Shane: No, but I think we’ve talked about all the things that you could use to define a survival metric, I think that’s the key of where the conversation was. But what you were looking for is give me some examples of survival metrics that give me clarity of why they are survival VIX versus a generic metric.

Murray: I think product survival metrics are boundaries. And the idea is that you empower the product team to make decisions quickly in uncertain situations by negotiating those boundaries ahead of time with stakeholders. And then you use data to track your progress against those boundaries so that you can see when you’re heading towards one and you can make a decision to stop pivot or invest. My problem is that these are not success metrics. I wouldn’t call them that. Boundaries are more like budget, time, scope. More typical project management things i think perhaps i’m wrong but that’s how i see it.

All right, Adam, I think it might be best to share some sites or blogs where people can read more about success metrics for themselves. What would you suggest? 

Adam: So there’s my column on built in where I talk about survival metrics. So built in.com. Just do a search for Adam Thomas or Survival Metrics. There’ll be a couple of articles that come up. Upcoming this year, depending on when this comes out. I’m redoing my blog, but there’s going to be a lot of tactical, practical, This is how a team came up with the metrics. These are the metrics, here’s how they affected them, I’m about finished with the first article in the series around a decision I helped the team with when it relates to building HR software.

And that’ll be out shortly, probably by the time this is out. That’ll. And so on my blog the adam thomas.com you’ll be able to find the blog where there’s a bunch of survival metrics, practical tactical stuff that you can read and dig into how this works. 

Murray: Hey, we really appreciate you coming on Adam. And, people should go and have a look at Adam’s articles on builtin.com about survival Matrix.

Adam: Indeed. Thank you gentlemen for having me. This was a blast. I really appreciated the challenge , and the, questions.