Join Shane Gibson as he chats with Justin Bauer on his experience combining the worlds of agile and product in a data driven company.

Read along you will

Shane Gibson: Welcome to the “AgileData” podcast. I’m Shane Gibson.

Justin Bauer: I’m Justin Bauer.

Shane Gibson: Hey, Justin, thank you for coming on the show. I’m really looking forward to this one. Before we kick off, why don’t we have a bit of a background about you and where you’ve been and what you’ve done in the world of agile and data?

Justin Bauer: Yeah, for sure. Thank you for having me, Shane. Definitely appreciate it. So, currently, I’m a chief product officer at ““Amplitude””. We help companies build better products through understanding user behavior, we’ve got a product analytics solution, digital analytics platform, so I get to spend a lot of time working with a number of our customers actually helping them be agile and ensure that we’re building products that customers love. And I’ve been doing this myself for over a decade of “Amplitude” for seven half years prior to that I actually ran a mobile gaming company, where velocity actually was one of our competitive advantages, differentiator for us the fact that we’re constantly learning with customers building new capabilities, I think that was really critical. And prior to that, it’s actually a digital analytics company, we’re helping a number of fortune 500 companies actually think about what their mobile strategy should be in this new way of building. So something that agile velocity, adaptiveness, has been a key part of actually my entire product journey, I think something I’ve been really focused on helping myself and other companies accomplish.

Shane Gibson: I think when I first started my journey, I was focused on combining the agile practices and patterns with the way we work with data. And the last couple of years, what I’ve seen is actually a lot of the things we need to adopt will come out of the product world. So it’s really a combination of product patterns, data patterns, and their job patterns to give where we get the really sweet spot on there. Everybody’s got a Venn diagram. That’s my new Venn diagram. How are you seeing that? Because obviously, from a product point of view, your product will know and for powering product companies and powering them with data and analytics, so they can see what success looks like, how does it work? How do you see companies adopt that data and that matrix to figure out what to build next? What’s working, what’s not working, getting rid of stuff, if they ever actually do get rid of features, because I’ve never speak to any of them?

Justin Bauer: That is the hope, “More is not always better”. I think it’s a really exciting time to be a product leader, because we have the ability now to truly understand what users want. They’re telling us through their interactions through their data. If they love a feature, love and new functionality, love your product, you’ll see they’ll come back time and time again. And if they don’t, if it’s confusing, you’ll see them struggling, you’ll see them not coming back that data is an important signal to you that you can use to improve that and this really wasn’t possible in the old days of building product when you shipped software in a box and delivered to someone. You actually didn’t know, you can take surveys. And you could ask them, but I’ll tell you what people say is definitely different than what they do. What they do truly is the barometer of actual product adoption and success. So I feel like the best companies realize this, there’s a treasure trove of information there for them, they can take advantage of to actually build and iterate and improve their product. But that also comes with a bunch of change in terms of how people do it. It’s a new thing. It’s no longer about this vision that you deliver for on high and say, “Go deliver this”, it’s now this world where it’s actually a lot of things aren’t working. They’re not necessarily, but you’re iterating quickly, you’re getting that feedback super-fast in real time for many companies. And that allows you to get something a lot better out there. So to me, it’s just a really exciting movement. And I think it’s the right way to build product because you’re building and the data is telling you what’s working and what’s not.

Shane Gibson: So one of the challenges, obviously, is that the volume of data is now so much bigger. We’ve been through the Big Data do bollock wave. But now we actually got to a stage where volume of data is a problem, not from a storage and compute because we’ve got cloud capacities now to just eat that stuff. But our ability to look at that data to understand what’s important to get that signal out of that noise. And with your background from gaming companies, where again, there’s a hell of a lot of user interaction data coming. And what you’re seeing people do in the product metric space, how are people dealing with it? How are they dealing with it? We’re no longer just looking at customer orders product. We’re looking at user directed with this feature five times quickly or slowly on a Monday but not a Sunday, or like their friends or not like their friends. Have you seen good patterns that people can use to really figure out what’s important to look at and what they can ignore?

Justin Bauer: For sure. The first thing is, you’re going to drown in the sea of data if you try to measure everything and necessarily understand everything from the beginning. So you got to be focused, what we recommend for people is figure out what your north star metric is that you’re really trying to drive. People are probably familiar with the concept of a north star. That’s like the direction you’re going the guiding light, what’s the metric that actually reflects that you’re actually going in that direction? I’ve found that when you actually ask leaders, that question and an organization, what is your north star metric, you don’t always get the same answer back. And so you should actually start there and drive that level of alignment, what is the most important thing that you should be driving for your customers today? And how are you actually measuring that get that aligned, because then what happens is now once you’ve driven alignment around that, then you can start to dig into the data to understand what are the drivers of that metric. So an example like for “Amplitude”. So for “Amplitude”, our North Star metric is what we call a weekly learning user. So that is an individual who’s using our product, they get to some level of insight where they save a chart, a dashboard report, and then they share that with multiple people, and at least two other people consume it. So to us, that is like a reflection of they’ve gotten to an insight they’ve delivered value, and is our north star. And it took a lot of actual iteration conversation to align around that being the most important thing that we want to measure. But once we have that, we can then start to break that down into what are the drivers of a weekly learning user? We got to have people, and the first two weeks actually activate. And then we had to come up with what’s our aha moment going to be? So then we had a whole bunch of debate as to what that should be? Once you get someone to aha, what does it mean to actually get them to the point where they maybe become a power user that actually starting to share? We create a definition of a weekly distributing user, which is sharing user and then, so you have a whole bunch of these different drivers, and then each of those have a whole bunch of questions. Is that the right way to measure it? What are the drivers of those? And then we can finally map our product work against that. So you’re working on building out the Slack integration? That’s a key part, your hypothesis is that we’ll increase distribution and people sharing within the product, let’s actually see if that’s the case, we can actually measure ourselves to see if that happens, or doesn’t happen. And what we’ve seen is in the instance, there with Slack, it definitely did. We had a couple of integrations where it didn’t, that’s great. We just decided not to push forward on that. Those aren’t things that we’re going to pursue. But now we’re aligned around metrics, using metrics actually guide us versus just falling prey to, there’s so many different metrics out there so many different things we can measure, I don’t know where to get started, have a hypothesis, have an opinion, and then help the data will help you actually see if that opinion is right or wrong?

Shane Gibson: How long do you reckon it would have taken you to go from the initial idea that you got to figure out what your north star metric is, and then to something where you’ve iterated enough time gone through enough hypothesis that you’ve got to something you’re comfortable with actually is an indicator of value for your customers and value for your organization?

Justin Bauer: The other thing is, it’s constantly changing because what you’re delivering to customers are always changing. If I think about an “Amplitude” first, our first North Star metric, which probably was maybe three months for us to get to this were much smaller organizational time was a weekly queering user kind of aligned with our mental models. So analyst’s product, so when queries make sense, check against that, that seems like valuable. And I think at that moment in time was probably the right thing. But then as our strategy evolved, as our user base evolved, initially was all about power users, they should be queried. But then we got to know actually, were more about product teams and collaboration. So now we care more about sharing that was then like, maybe we should actually measure this differently. And I think it probably took us maybe another three to six months of iteration, we tried out a bunch of different ways of measuring it, we looked at how those different North Star metrics maps to our customer retention, because we actually wanted something that was both customer value, but also business value. We did some regression analysis, like a bunch of different things to actually uncover that. Like I said, there was a bunch of iteration. And we’re constantly rethinking that now we’re a multi-product company, maybe we actually should be something different for us we’re exploring should actually be at a team level versus a user level. So the point is, you’re constantly having this conversation and making sure you’re confident that we’re, the key thing that you’re measuring actually is pointing in the right direction.

Shane Gibson: Sounds about like sprint planning, and that the value of sprint planning is the conversation by the team on what they’re going to do, that says valuable was actually locking in what they think they’re going to do this iteration and what they’re going to deliver. It sounds like the conversations you were having around what your North Star metric could be how you could break it down how you could measure it were as valuable as actually coming up with the metric itself?

Justin Bauer: Because it’s all about alignment. That’s the thing when I came back to like, it’s not a product leader alone makes that decision. It’s about actually alignment with the company. And that’s where I found that, actually, when we go into one of our customers, and talk to them about how do you come up with the North Star metric we actually find it’s a strategy conversation, not a metrics conversation, because they’re actually not aligned to the strategy. And then of course, you’re never going to be able to align around a metric if you’re actually not aligned on the strategy.

Shane Gibson: The other podcast I do that no nonsense when we hit Jason Yip, from Spotify and we were really keen to do down around the, what used to be called the Spotify model and what they’ve done. And he constantly came back to, it’s not a conversation about team structure, it’s a conversation about strategy. It always goes back to that. One of the mistakes I see people make with North Star metric. So as they think about it as an internal metric, and the key thing is from my understanding of it is it’s about value to the customer. If I think about our startup, very similar problem, which we haven’t solved to what you articulated, and that we’re a data platform. So we do that middle. But we don’t do the last mile, we don’t do the presentation of the data. Our internal metrics could be volume of data collected, or internal metric could be number of rules, when you’re effectively use the term rules for the transformation code, a number of rules have been created and executed. But none of those actually have value for our customers. The customer value is actually using their data and actually using it to make a business decision. And even though we don’t do that last mile, we empower the last mile. So that’s where we got to as we put our customer hat on, and what’s the value they give everything else is indicators that they’re not core measures, actually determine the North Star value. So have you seen that? Have you seen that lots of people focus on vanity measures or easy to measure or internal measures? And they’re a good start if you’ve got nothing else, but you really have to get to that understanding of value to your customer?

Justin Bauer: Exactly. I think the way you frame it is a good start. It’s not that those things don’t matter. And honestly, they’re probably going to be drivers, or check metrics. And we have both in how we think about our broader metrics framework, we have things that we understand our drivers that we’re really investing on. And we have some things that are check, which is we’re not currently investing in this. But if that fell significantly, you probably want to understand why. So she’s definitely be monitoring them. So many of those things are going to be part of that constellation of metrics that you’ll have. But to your point, they’re not going to be the North Star, because the North Star has to be a leading indicator of customer value and business value. And it really needs to be both, it has to be very well aligned there. And a lot of times those vanity metrics actually aren’t reflective of the customer value. And in those situations, it actually is worth spending the cycles to think about, can you get to that I remember for one of our products is an experimentation product that we have. And we’re really thinking like the whole point of this is not necessarily to run experiments, obviously, running experiments matters. But really, it’s about making a decision. At the end of the day, whether you decide to roll it out or not. It’s like you’ve made a decision. But we didn’t have anything that really measured, we didn’t know if customers are making a decision that actually started to get us thinking about well, why not that is part of the workflow here. And so we actually started to extend our workflow into that, and actually help people see you should make a decision here, don’t just let this sit forever, it’s gonna sit in your codebase. Like, now’s the time to make decision and make a decision faster. And then we started instrumenting that, and now we actually see how many times our customers are making decisions positively or negatively. And that’s a metric. And so now we actually have that as our North Star. And so we actually build our product in a direction that actually reflected that value. And then we can measure it, you maybe can’t do that with every time, but I think it’s an important conversation to have.

Shane Gibson: That aligns back to what your customers are looking for as well. They probably saying if they can actually measure how many hypothesis they had, how many experiments they ran, having the decisions they made, you’re helping them measure the value while measuring the value that you’ve offered. I’m old enough to have been through a few waves or errors of data technologies and ways of working. There’s been a lot of noise on LinkedIn. I remember, in the old days, we used to have old tools like business objects and Cognos, Oracle Discoverer. And we had a semantic metrics layer, we had a piece of capability between the visualization tools and the data itself to create a measure once and share it many times. And we’re starting to see that with modern data jingoistic, the idea of semantic layer matrix layer coming back, and as a cool capability, what are you seeing in that space? Are you seeing that capability of easy ways of defining metrics and sharing them as been where the markets going? Are you seeing something different?

Justin Bauer: I do. I think it makes sense. I think the mental model makes sense for folks. And the reason for it is because one of the benefits of modern tools like enable to another’s is that it’s increasing accessibility. But you think back to the old days of cognitive services, like it was really only accessed by analysts, whereas now we have a world where fully leaders, PM’s, marketers, they actually have access to this data themselves. So they can actually make decisions fast. And that enables speed and velocity and all the great things that you want with democratization. But the democracy concept is interesting when because then you gotta put guardrails on it. Because what ends up happening, and I’ve definitely seen this myself is you’ll have five different definitions of a metric. And so everyone thinks when we’re talking about number of video views, that we’re talking about the same, but actually, what does it mean to view a video take immediate examples is that they watched it for two seconds, they watched it for five seconds for a whole minute, there’ll be lots of different definitions of that. And so you do need a line around those definitions. You’re all speaking the same language. It also makes it a lot easier for people to be able to then build on top of that, that’s the thing I’m actually like the alignment and then the building blocks. I don’t want to just understand how many video views like that might be some vanity metric but I to know if people do five video views in their first seven days, are they more likely to retain? Now, I’m starting to understand drivers of my business. But then if I can just pull, this is our definition of what a video view is. And I can start to build my analysis on top of that, and not be so reliant on a data scientist or analysts do that, but I used to do that myself as a product leader. Now, I’m able to get to that hypothesis validation a lot faster. So it’s an important direction because of those concepts, not necessarily because of computational speed, which maybe was some of the reasons why those things existed in the early days, it’s actually an alignment, and collaboration are the reasons why it’s coming back.

Shane Gibson: I agree. I think we got the metrics layer in the old days, because we effectively put an In-memory OLAP cube behind it to give us performance because we couldn’t handle the data on prem databases. We had to be interesting to see what we get now it really is just true, a true semantic definition, or whether we still have some caching technology that adds value. And you raised interesting point there, which we often forget about is that, like the North Star metric, the defining of those metrics starts off simple. And then we answer the first question, and then we’ll get given the next question. So how many customers logged on? What feature did they use? And then, time between features or clustering of cohorts of users by feature? When teams starting off, they tend to want to boil the ocean, they tend to want to go and get all the questions that may need to be answered, and then try and build a data that supports every question. And I’m a great fan of just starting small using agile iterative process, which is just go and say how many people logged on? That’s a good metric, not your North Star but it’s a good thing to monitor, it’s good thing to be able to answer that question. And wait for the next question within iterate it out. Still see people getting a big requirements catalog of everything they need to build, and then still spending too much before they get any valuable feedback from their stakeholders.

Justin Bauer: Unfortunately, we do still see that. I also think that we can probably say quantitatively, that approach tends to fail more often than it succeeds. When you start from this huge waterfall, these are all the things I want to be able to measure, because you’re not delivering value soon. So it’s all the same agile concepts that we think about with software development, it’s honestly true for any project as well, which is get a couple of wins early. So start with a key business problem that you’re trying to better understand and solve for. And then think about based on that, how would you measure success and then lead with some hypotheses that you have on what the drivers of that are, and then use that to come up with a very basic tracking plan, and then put that into practice. And what we found is that if you do that, then you can probably get up and running in a couple of weeks, and actually literally have real data, validating your responses. Whereas if you take the approach of let me figure out every single measure that anyone cares about, and what’s the perfect thing, and then we’re going to instrument, every single one of our features that can take years. And, unfortunately, in the period of years, you’ve now gotten further behind your competitors who do understand these things, you’re not delivering customer value. And so we very much preach, start smaller, deliver value, thinly, slice deliver value much faster to your business and your customers by figuring out a couple key things prove that you can do it, and then think about how to scale.

Shane Gibson: I think one of the challenges is actually on the technology side these days. In the old days, we’d have Change Data Capture technology, we’d go and hook into the production database, and we get every change of every record from every table that would come across into the Data Warehouse. And that enabled us not to worry, we worry about storage, we worry about changes the schema on the source system side. But we never had to worry about actually getting the data because we knew if the data was updated, they got sent to us. Now in terms of product instrumentation, actually, there is effort to capture those events are the app developer has to actually say these are the events that are important. We want to capture them. And we want to move them to the analytics platform so we can query them. So how do you see people dealing with that? How do you see them deciding what events are important right now, and the fact that when they add a new event in the future, they don’t automatically get the history over all time on how that a feature or event was used since it actually changed the way people think about instrumenting the collection of that data or not?

Justin Bauer: I think, it encouraged people to think more about the business value first, because it isn’t this world of just collect everything and then we’ll deal with it later, like the concept of the old data lake. I actually think it’s a healthy direction, because honestly, in a world where you collect everything, it ends up being a big mess. And you honestly don’t get a lot of value out of that it’s a number of transformations. Those things aren’t production quality. I don’t know if we actually got what the promise was there versus in this world where you’re much more explicit about what you want to track, you’re much more likely to think ahead of time of what is the business reason for this, and then you’ll actually get that out of it. So there’s a little bit of effort. I think it’s not really use that much though in the grand scheme of things, if you actually are thinking ahead of time as to what is it that you really want to understand. And to me, that’s should just be integrated into the software development lifecycle. Like everything that we’re shipping is a hypothesis. It’s a hypothesis that this is going to deliver a better experience to the customer. Most things actually don’t that’s the sad truth of it. And so if you think of everything as a hypothesis, what we call them bets, everything is a bet, then how would you not have a measurement of understanding that just be embedded into how you build it? And so to us, it’s like a simple just next step of the process is great. So why are we building this feature, every engineer understands that while they’re building it, we want to understand this, we think this is the business impact, here’s how we’ll measure it. And so once I get to the stage of that, I can put in the simple tracking code to see that. And then once it’s released, that’s the first thing we’re looking at to actually see that impact. So when you integrate it into the development lifecycle, and as expected part of the process, it actually isn’t really that much extra lift.

Shane Gibson: I think one of the differences between, say enterprise applications is they’re the only time you ever see a field come off a screen an enterprise application, ERP, or whatever it is, when you do one of those horrible upgrades or take your six months, and you lose half your features. Whereas in the product world, in theory, we’re losing things that don’t have value on a regular basis. So therefore, events that we’re getting just disappear, because that feature has disappeared from the product. How often does that happen? Do you get any visibility on how often people actually remove features out of their products?

Justin Bauer: That’s a good question. So we probably could actually have a monitoring feature where you let our customers know that we monitor and alert to them when data drops off. And so that is actually a really great idea for our next product report, we might actually see if we could pull a global stat and how often that’s happening across our customers. I might chat with the team about that.

Shane Gibson: We don’t want to do it, somebody must be using it, I was gonna hit that. We made a bit we put effort in.

That’s why we call them bets. The mental model of that of just recognizing that there are hypotheses not everything’s gonna work, that’s okay. Part of it, I think that organizations have to be comfortable with is that’s okay, that’s an expected part of it. And not everything is going to work. Because I think a lot of it comes back to the ego of the person who built it, and they don’t want to let go. But at the end of the day, it’s not about serving our ego is about serving our customers and the customers, it’s complicated the user experience, and that is not serving your customers.

Shane Gibson: I suppose it relates also to the cost of the ability to add a feature. So if I can add a feature really quickly and easily and I don’t have to go through a whole lot of governance forums and fight for that feature, and get it prioritized at the top of the list. If I have the ability to edit fairly easily, then those bits are cheaper, I’m spending less money on that bit. Whereas if I’m invested, and it’s taken me six months to get it to the top of the roadmap, and then finally that thing turns up, it’s going to become more precious to me the amount of time and effort.

Justin Bauer: For sure. It’s literally cost fallacy.

Shane Gibson: Cost of change is important. It’s important that got me back to that. So then it must be important that when you are defining these features that you want to add to the product, that actually you also define measure of success from a data point of view, because how are you actually going to measure? And that’s hard. We know that even corporate metrics are hard. We have simple ones, like profitability and revenue and margin and those ones, but it must be hard to actually for people to think about if I add this feature to the product, how do I measure success? How do I measure some data that’s going to tell me keep it or kill it?

Justin Bauer: So it’s hard, but it’s important. I think that goes back to having that constellation, and ensuring that everyone at every level within your organization understands, what’s the broader strategy, and then how you measure it? And so that these things kind of work in parallel? You can it’s able to actually do this, what’s our overall product strategy, which obviously layers up to the company strategy? And then within each product, what is their strategy? And then so overall, North Star metric driver’s products have theirs and then also North Star metric drivers. And then it just continues to layer down from product to pillar to pod that’s our organizational structure. And then within a pod, if you’re working on a bet, so that is a hypothesis. So it has a clear sense of what the impact is. And then anyone working on that can actually see why am I working on this, there’s a hypothesis that if I build this thing, or change this behavior, it will ladder up to my pillar level strategy, which ladders up to my product level strategy ladders up to the overall strategy, you can see that and you can see all the cascading metrics that go with that. If you have that framework in place, then it actually becomes a lot easier, because then I know that I’m on this team whose job it is to try to get people to consume more. That’s because I’m part of this group that’s focused on how many people are consuming in the product. I have a hypothesis that if I build this thing, it will get people to consume more. What’s the first thing I want to measure? Do people use it because if they’re not using it, it doesn’t but then two people who use it end up consuming more? Now, I’ve because I have this broader framework. I actually understand what my role and purpose is within it. It makes a lot easier than figuring out what are the  things to be measuring there versus if you don’t have that they’re not aligned then they don’t actually understand how they fit within the broader system.

Shane Gibson: And then the challenge is making sure that the matrix for each of the pods is enough to find that they can see the success. But then you do know it goes up into the higher level metric, because there’s always that cause and effect, it’s everybody else is making beats. So who actually made the dial move that a little bit to the right is always a hard one?

Justin Bauer: Yes, that’s hard. And then the other hard part of it too is this is a living and breathing system. They’re all hypotheses in every layer, the fact that consumption matters, is a hypothesis. So you have a team trying to drive consumption, but it may be that actually consumption doesn’t matter. So you as the leader needs to be testing that hypothesis. And actually, you can use metrics to do that. I’ve definitely had examples where we’ve seen one team just do an amazing job driving growth in their area, their goal was to drive maybe 100% growth, and they more than doubled and tripled it, that’s awesome. They did an amazing job to tripling that actually lead to a higher level impact above that. And if it didn’t, that’s not on that team, because they did their job that’s on you. I’ve now demonstrated that maybe this isn’t as important as I thought, maybe I should shift the strategy here. And so metrics can help give you that feedback mechanism. Like I said, it’s a whole learning system.

Shane Gibson: It doesn’t have to be 100% quantitative. So we know that if we want to get a good estimate, what we do is we get a bunch of people to estimate it together. Because we know as a group, based on the conversations and the way we estimate a group of people are always going to come out with a better estimate than one person, even if that person is an expert, I’m assuming from a metric point of view, it’s the same thing. Every team has a set of metrics that are aligned, and they’re moving the metric into a better space, then by default, the whole organization should be moving forward, because we’ve got that alignment of every pot moving forward slowly or quickly. And so therefore, in theory, the organization’s moving forward, it doesn’t have to be 100%, data driven and alive, it’d be great if it could, and we get the , the whole cause of cause and effect. And that’s quantitative and statistically proven but often that’s hard.

Justin Bauer: It is. And I think sometimes people fall trap to wanting to prove out the causal relationship of everything when it is in many places not possible, because there just are certain things you can’t capture, and represent within the data. And so that alignment is really important. There’s a classic example from Facebook of the early days, which is their goal was to get seven friends within the first 10 days. And that was a big driver of growth for them. Now, was it exactly seven friends and exactly 10 days, could it have been eight friends and 11 days, that level of precision is not the most important thing is the fact they were all aligned on that because there was a lot of confusion as to what we should really be focused in on at that day and that gave them that alignment. And that was good enough to be great. We can actually see if we’re driving this, we can see how many friends someone has and how quickly they get to that that alignment piece is a really critical one. And then later on, you can measure to see if actually you want to shift it you can but the fact you’ve got a whole org aligned around driving something is pretty powerful.

Shane Gibson: It’s just coming back to that team topology talked about organization topology, I’ve gone down to panels and pod. So as a pod self-sufficient, do they have all the skills they need to be able to ship something?

Justin Bauer: That’s right. So PM design engineering, there might be probably like three to five engineers within it. It’s the unit that allows us to actually deliver, and so that’s our base level unit.

Shane Gibson: One of the things I’m doing is I’m doing a referral program around product, because it’s a skill that I’ve dabbled with. By going through this, one of the things that’s been really interesting for me is the team topology and product. Because the way it’s described in the course materials very much around a product manager, and product analysts that the analysis piece is actually embedded into the development team is not at the end of the supply. Does that common that actually you would typically have data savvy analyst in the team that’s actually building all those features?

Justin Bauer: I think teams decide at what level they want to have that type of analyst support. And I think that actually is a reflection of how empowered and enabled the PM’s are with data at a company like “Amplitude”, where we obviously use our own product solution. As you can imagine, our PM’s are highly enabled with data. And so we don’t have to have an analyst on every single pot. But it does make sense to have an analyst support because of some of the things that you talked about it it’s not about doing the analysis, it’s actually about the upfront, how are we going to measure these things? So the conversation around what a North Star metric is, an analyst plays a critical role in that because that’s actually something they’re really good at. They’re really good at translating what the business strategy is to actual metric. They’ll understand what can actually be measured within the product, product analyst and “Amplitude” will help them put in practice, the processes, the best practices, we have to ensure that’s happening. They’re part of the governance count to ensure that we’re actually managing and maintaining data quality over time you want to serve those jobs. And then I think you have to think about where you’re at within your mature already as to then how much do you need an analyst do that versus how much can you put on a product manager, I’d say we’re fairly mature in this. And so that means we’ve built enough systems, processes and software, so that we don’t have to have so many analysts. But if you’re pretty early in that journey, you actually want to have more just to make sure it gets going. But hopefully, you can actually systematize that stuff and use software. So you don’t have to put like a data scientist, every single team that can be pretty expensive.

Shane Gibson: And is it part of the maturity, you’re typically now hiring product managers and product leaders who have a data background as well, or do you have a literacy programs, so when they come in then up-skilling them in some of those areas that they’re not getting in their previous roles?

Justin Bauer: Because there’s so many different types of product people out there, I actually think it’s really important to think about what’s the right product fit to your type of product. And so we’re in a unique position in that given that we’re an analytics company. By definition, I’m going to hire people who are familiar with analytics, because it’s what we build. But at a gaming company, I’m gonna hire PM’s who are very familiar with analytics, because it just drives so many of the decisions, that may not be true at every company. There’s a level of sufficiency that I think every product person needs to have. And that’s true for every skill set design and customer research, analytics, business technology strategy, all those things are different capabilities of a product manager. Some companies you need to be super spiky in one areas, but less than others. And “Amplitude”, people need to understand their lives because it’s what we sell. We then do still have sufficiently training like how do you actually get embedded and use “Amplitude” and stuff like that still exists. And I think that’s important, because it is a critical part of the skill set. And maybe that wasn’t the case 10 or 15 years ago, but it definitely is today is an important part of how people make decisions.

Shane Gibson: We talk about maturity and people’s T skills. I talk about novice practitioner, experiencing coach. I think it’s important to actually understand maturity of the organization, both in terms of product and data and agile. So in terms of they start off and things are raw, things are funky, because you’re all learning. But after a while, you get to live on maturity, and then you need to change some things, everything always needs to change and needs to adapt for that

Justin Bauer: I actually find one of the ways you can actually change your own organization’s maturity is actually hiring someone in who is a level higher that. So if you feel like you need to improve on how you actually embed analytics into how you make decisions, go find one team that is moves faster, you think there’s an opportunity to do that, and go hire a product manager who’s really good at that. And you’ll actually find that they’ll be successful, and they’ll start other people will see it, and you’ll start to get a more bottoms up movement there. So actually I think it’s a great way to actually improve maturity institute like that.

Shane Gibson: We see that a lot with organizations, we see team and an organization start to experiment with agile and get really good at it. And then it becomes a bit like a virus and what else is going well, they look like they’re delivering they look like they’re having more fun. How can we do that too? Unfortunately, some organizations, that’s when they then stop it and bring in my safe consultants and transform that whereas it’s growing, just let it grow as a good virus, good mold. So coming back to that idea associated with so as we get more and more self-service, more and more distributed work. We end up with a governance problem. And in the past, we talked about centralized governance, and it’s been a nightmare. It’s been a bunch of committees, a bunch of people who don’t have time to do the work, getting a bunch of decisions were bugger all context and then making some decisions that everybody else has made to follow. And what do you see now in terms of is a theory coming out of federated governance, but very little patterns. It’s just like words. So what do you see in terms of the idea of governing around self-service, do you see anything useful out there?

Justin Bauer: I’m definitely a fan of that movement. I think we’re seeing that within our customers for ourselves as well. I think it’s about really lightweight governance. And there are just some high level best practices like naming standards, let’s align around that and start small. I think the failure point is you describe I’ve seen a lot of companies as they have these very strict governance was like we have this global schema. And one needs to confine to that. And what will happen is that if something doesn’t confine to it, they’ll go through the process once of trying to get in front of the agenda of the schema Council, and then the council will be like,  or no, and we’ll take forever, and then everyone else will see that kind of to your point. This is the bad part, everyone else will see how painful that was. And they’ll just be like that I’m gonna go and do it myself. And so then you end up with a whole bunch of different standards, and you’ve created even more of a in so instead, if you can keep it super lightweight, just start at the naming convention. And we’ll agree, a simple thing like video played should have one way to name it. Start there really lightweight and see if you can drive alignment. Second piece is identify a couple of the key things that drive a business strategy forward. That are a reason why you need alignment. So we’re trying to better understand how certain products as part of a broader portfolio are driving network retention for us, because we’re now multi Product Company and we need to see like this is a starter product. First, we want to see how people navigate between them. Everyone agrees it’s a company level strategy. Use that to define. So we’re gonna have a golden set of tables that we really want that enable that to happen. Everyone needs to understand this, attach it to what matters for business help people understand the why. So in these certain areas fit within the framework. I think the other thing that’s really valuable is this concept of golden table. And then there’s just a place where you just experiment run and just know it’s going to be messy. And you just let that be recognized that’s the case, and allow people the opportunity then to clean it up but that’s just there. There’s like playground, that’s cool. Let them move fast. Let them instrument, however, they want. And then if you find there’s something in there that actually is really important that’s needed for a business perspective, you move that into then the concept of the golden table, and I think teams will get that they’re like, I understand other teams are making decision on this. And so we’ll make sure we fit to the standard there but you’re still giving people the ability to instrument quickly measure the things that they care about. I think if you start that lightweight, only the most important things for the business you can start.

Shane Gibson: I think the idea of focus is really important. I remember in a previous life, Shawn Mugger was working with us. And he did a PhD around Procurement Fraud. So he got a whole lot of open data from a fairly large country. And as PhD and again, Shawn, if you’re listening, and I’ve misunderstood what it was, but as such, government actually published all the procurement data people bought when the procurement team was put out having one of the coolest stuff. And what he found when he looked at the data was in areas that were large tanks, ships, planes, there was very little fraudulent behavior. Because all the different government agencies were really focused on each other. They kept each other to account from memory, photocopiers, and printers and things like that. It was right. So, again, it comes back to the idea of governance that if there are things that are important, and you make them visible, then actually teams will hold themselves to account that your answer around naming standards. If you actually say these naming standards are important and then you publish metrics that say, the schemer over here is not compliant with a naming standard, then if it is truly important, then other teams will have their conversation because they’re dependent on that naming standard. They’ve written code or interfaces that say we understand how the naming standard works. And we’re going to rely on it as a contract versus going back to your parents, your brothers and sisters or you don’t account and they tended on your parents always got way too busy, where everybody and could never figure out what the real truth was. And I liken that to governance, as teams, we can hold each other to account for the things that are important and things with agree, but as long as it doesn’t slow us down too much.

Justin Bauer: If the agreement that so that’s the critical part, because everyone can understand everyone’s lived through the pain. And so collectively, you’ve made a contract with each other to follow through on that. And that’s where it starts small. Get those maybe that will get to the top 10 things. But if you start from 10 things that people don’t understand why that matters, they’re not going to follow it because it’s going to slow them down. And therefore, they’re going to find the shortest path actually delivering what they need to.

Shane Gibson: And again, it comes back to a cost to change. Sometimes if it’s too hard to change in the beginning, then you won’t change and then that’s a fail, because you’re not able to make those bids, you can’t make 100 small bets, you can make one big bet. And then you double down on that bet when it becomes an expensive and dangerous because you can’t fail anymore, because all your eggs are in their basket. So coming back to this idea of product versus agile, and I’m using the old finger quotes. When you talk to product teams, you see a lot of agile behavior, you see them inspecting their work, adapting the way they work, and making experiments will be getting value out to the customers as early as possible. You see all the things that we can use to describe agile iterations where there’s a time boxing of stuff. What you don’t tend to see those all the Agile words tend to hear Scrum and Scrum coach, here all the things that we hear from an enterprise point of view is that what you found is that the behaviors there but the terminology and the words aren’t was, am I just in the wrong place?

Justin Bauer: I think you’re right. So I’m based in San Francisco, so definitely part of the Silicon Valley group. And we definitely do not use those words. I think, in many ways, because any concept, people take it to an extreme and it becomes a trained way. And there’s only one approach. And so I think we’ve gotten away from that a bit. Because if you use the word scrum master, people have this strict assumption of what that actually means versus the principles behind it. And I think one of the keys to being successful is be principle oriented, but be flexible on what the process is, the process might change and so many ways for many companies, I think in the valley, we believe in the principles but actually how we how that might get executed will look differently because there are different things, people were building different products and building for different customers in different industries. I think that has been one of the challenges with agile, I think this is true of any big movement. And this is the only way to do it. This is the one way and there’s actually lots of ways to do it. And so it’s much more around the principles and then figuring out what that means for your organization.

Shane Gibson: You’re right. We get a gospel that we have to follow now and don’t go outside the boundaries, even if it looks like going outside them has value to you. And we get terminology bleed, the one that I still struggle with as the definition or the difference between a Scrum Product Owner and Product Manager from a product company, because they look close enough to been the same, but they’re not, they do fundamentally work in a different way. Have you seen that blurring of terminology?

Justin Bauer: We definitely have in many ways, it’s moved back to there as a product manager in terms of we’ve got product management, we have product operations, which is interesting. I think that’s a new kind of new field that’s growing. And there’s a key partnership between those folks. And so that has been a blurring of traditional roles that were owned by product owner versus product manager and kind of in traditional sense, like what that means. And then the other thing that what I have found is that it can differ depending on company, the most important thing is there’s clarity within the organization as to what the role and responsibilities are. But they can actually shift in some organizations, you might have an outbound PM, which is actually much more like a product marketer and an inbound PM. And you might have in other organizations, you might have a PM, technical PM, somebody might have pm product operations, maybe what we’ll find 10 years from now or 15 years from now is actually we do align around all of these definitions and come to a standard. But at this stage, at least, the most important thing is that you’re very clear what the roles do within your organization’s you’re hiring people that Matt mentioned that.

Shane Gibson: I think in 10 or 15 years, we’ll have a new buzzword, and hopefully it’s one that has value and the good patterns and it’s not just a buzzword. But speaking a buzzword, so product box is really interesting to me. I think we’ll get, DevOps, RevOps at some stage. But ProductOps is quite interesting for me, because I can’t tell if it’s more of a DevOps thing around automation of platforms and processes, or if it’s more way of working around how you determine what features are the highest value and what you should develop next. So from your point of view, how do you differentiate between product management and product operations in your organization?

Justin Bauer: Our mission for our product operations team is to empower all teams to execute across the product lifecycle, with clarity through processes, tools, insights and communicate. And so what that means is that, product operations and service of the broader organizations not just in service and product management, that’s really important. It’s in service of the broader organization. It cuts across the entire product lifecycle. So everything from the research phases to post launch is focused on improving velocity and alignment. And then it’s how is it was the owner of what are the processes? What are the tools are the ways that we capture insights and how we communicate between these different parts of the organization? Probably the best analogy to this is, if you think of the product team is building the ship, product ops builds the shipyard.

Shane Gibson: Again, I’m a great fan of patterns. I try and determine which patterns somebody falls into. And so these two patterns are just heard within the one is this idea of coaching circle. So this idea that we have agile coaches who look at delivery teams, and help them iterate their way of working, have a system to go, where’s the blockers, where’s the things we should change, what you should change and how you’re going to do it. So product ops, I heard some of them standing outside of the workflow, outside of that way of working, observing and then helping everybody go, this is not working, what are we going to do to iterate that way of working? And then I did actually still here some platforming ops, DevOps stuff in terms of some automations and removing the dropsy work from those teams, because they should be focused on the valuable stuff and we should automate what we can with the machine. So it’s a combination of those two, from your point of view to get there. It makes sense. We’ve covered a lot in just under an hour. So that’s been great. I love the way we’ve just bounced around in so many different areas. But of course, we had patterns coming out that seem to be common across both the product world and the nonprofit world, shall we call it. So if people wanted to get ahold of you and hear more about what you’re doing and find out more about your space with this way for them to get ahold of you?

Justin Bauer: For sure. Twitter, justinjbauer, as my Twitter handle. I’m on LinkedIn quite a bit too. So people can just find me Justin Bauer and follow me on LinkedIn, and then my company as well. So Amplitude, we obviously write a lot about these types of topics as well. So that’s a really great resource for people to fall into.

Shane Gibson: This is one of the things I love about the new modern world of product innovators, that large companies are much more open to sharing now with your teams, publishing great content will stuff that it’s actually usable and making it freely available these days. So I think that’s one of the benefits we’ve got out of this kind of modern way of working.

Justin Bauer: We believe it’s important, because in many ways our mission is to help companies build better products through data. That’s a transformational change. We understand that. So it’s not just about us building software, we’re trying to help people understand what are the processes that people take to do that. Get those stories out there from companies who have successfully done that. So we’re big believers in pushing that thought leadership as an organization.

Shane Gibson: It’s always saying sharing is caring as well. It’s good to have that information out there for people who need it. Thank you for your time. It’s been awesome and that will catch everybody later.

Justin Bauer: Great, thank you.