DevOps with Mirco Hering

Read along you will

Shane: Welcome to the No Nonsense Agile Podcast. I’m Shane Gibson.

Murray: And I’m Murray Robinson. 

Mirco: And I’m Mirco Herring.

Murray: Hi Mirco, can you tell us a little bit about who you are, what your experience is, and why you’re here talking to us about devOps? 

Mirco: Yeah, absolutely. I’m Mirko Herring. I’m the managing director with Accenture. And I’m leading our global DevOps practice. But locally I’m looking after our modern engineering practice. And that’s where we bring agile DevOps, quality engineering together to really help find better ways of delivering solutions to the problems that our clients have. 

Murray: And you’ve written a book about devOps, haven’t you? 

Mirco: I have. It’s DevOps for the Modern Enterprise. I didn’t like many of the DevOps books in the industry because they’re very preachy. And what I wanted to do is write a book where every chapter tells you what you can do yourself. 

Murray: So when I see people talking about DevOps, they have a very strong tools focus. And I don’t think that’s what DevOps is supposed to be, according to your book. So could you give us a definition? 

Mirco: For me, Agile is a lot about building better solutions and then DevOps is the way to build the solution in a better way. It’s not just a tool. It’s not just a continuous delivery pipeline. It’s not just bringing dev and ops team into the same room. It’s really looking at the end to end engineering. 

So I think one of the key things when we start talking about challenges is that it’s not about the target, it’s really about the journey.

Shane: So it sounds like Agile, where we go Agile’s not scrum. It’s a whole lot of things. It’s, iterate inspect and adapt . It’s people working together. It’s looking at ways of making things better in the way you work.

Mirco: There’s a significant overlap between Agile and DevOps and they suffer from the same problems.

Murray: Would you say that DevOps was begun by Dave Farley and Jez Humble when they wrote their continuous delivery book? Is that where it has its roots?

Mirco: So I think it’s Jez Humble, It’s Dave, it’s Gene Kim, who I think is the flag bearer at the moment with his Enterprise DevOps Summit. I think it started with Amazon that can do a thousand deployments a day and Jez’s book behind that. Then it ended up being about the continuous delivery pipelines and then became Docker. 

The challenge that DevOps has is all these vendors that immediately hooked onto it. If you had a config tool before, then now it’s a DevOps tool. If you had a testing tool before, now you have a DevOps tool. Everyone just rebranded. Everyone, DevOps washed their tool set. And that didn’t help us to be honest, because all of a sudden everyone had DevOps.

When I go to my clients, I’m always saying the tools are actually not going to make a difference. I don’t actually really care that much because they’re not that different. It’s a matter of how you use them and that they’re appropriate for the problems you’re trying to solve. 

And there’s just so many tools to talk about. In the DevOps space, you can go to a conference and every talk will tell you about a different tool, a different programming technique, a different architecture, and you can go nuts with it. What I find frustrating is that not a lot of people talk about what difference it makes.

And so if you ask someone, they have adopted DevOps, they will talk to you about the number of deployments they did or what tools they’re using. But it’s very hard for them to describe what difference did that make, are you actually getting faster or is the quality getting better? Are you getting a different product out? And they very often struggle with describing that.

Murray: What is the problem that DevOps is trying to solve?

Mirco: I, like you guys, have been a developer in the past. And to me, the problem that DevOps solved is that I can focus more of my energy building solutions. So there’s less toil, there’s less things that I have to manually repetitive. There’s less flow back from production on defects that I have to deal with or outages or any of those kinds of things so that i can spend my time on the creative part, which is solving business problems with my IT skills. And that is what DevOps should do. It should maximize the productivity of a developer and minimize all the other stuff. 

Murray: I always felt like DevOps was supposed to be about the developers taking responsibility for the operations. Bringing operations and dev into the same team. So you’re building a product, you have to support it. And that also means that you have to build something that is easy to support and easy to update. So then you go and automate all of your environments and turn your environments into code. And you also get continuous feedback from users for performance issues and you feed that back into your product development cycle. 

Mirco: The last bit you said there is to me, one of the key things that has come up now that people start to realize, because that feedback cycle that you’re getting from production, not just on defects but is this feature being used? Is this specific component of my solution working? That feedback should feed back into the product management cycle. And so few organizations do that. 

Shane: How often does it happen? Because the one thing I hardly see, especially in the data space, is that feedback loop. We will deploy data platforms, and a lot of teams will never actually record who’s using the data. How often, what problems, and we’re data people, so we deploy data and then we don’t use the data to see if we’ve been successful. Is that the same in the DevOps world? 

Mirco: 100%. there’s a lot of vanity metrics in DevOps that people use. I had a very interesting conversation with Jez Humble when he was here in Australia visiting some of the banks here. There’s no way you can tell me that these hundreds of deployments a day actually changing functionality because that would be pretty incomprehensible for people using the product. And he’s like, yeah, you’re absolutely right. Let’s say we do 600 deployments a day, 590 of them does not carry any new functionality to the customer. It’s a good healthy practice, but it doesn’t tell me whether the payload on that is useful or not. Have we used our, highly optimized, super fast delivery mechanism to build good stuff or just to update metadata? I want to get to, I’ve built a feature. I’ve put it very quickly into production and I now can see that people interact with it. I would say 80 percent of the organizations don’t even measure the usage of the feature. And then we can talk about, do they actually get value from it? 

Murray: Well, there’s all that research about how much of a product’s features people use. And the Standish Chaos people found repeatedly that about 70 percent of the features we build for people are hardly or ever used. And that’s where product managers and product owners come in. But as it turns out, they just guessing if we can put little things out and try them, then we can get some measurements on whether people actually want them 

Mirco: And this is where we bring this back to end to end teams that actually own the product. Because if I’m only a development shop, I’m going to get evaluated by how quickly do I translate requirements into code. And I don’t know what that means other than I’ve shipped the thing. If you are an ops organization, then you get measured by how much is this thing available, whether or not it’s useful. It’s different. If I know how to bring these two things together, then you start feeling real ownership for it. 

I’m not implementing a requirement. I’m trying to solve a customer problem. And now when I put this into production, I want to understand how much of the problem have I solved so that I can then know, do I continue to invest in this or do I go somewhere else? And you can’t do that when you have all these different handovers and you create documentation in between these different steps.

When we do DevOps right, we are optimizing visibility into end to end delivery process and the feedback. Now you can then say that’s basically agile, isn’t it? And I would say yes, it’s just from a DevOps perspective, we don’t care whether it’s scrum or SAFE or Kanban or whatever. 

I was sitting in an agile training in like 2009 and they talked about these sprints and that was a new concept for me at that point. There’s no way I can do like a weekly sprint if I don’t have test automation, if I don’t have automated deployments. Because there’s just so much effort that I would’ve to put into this stuff. In that moment I was like well, I’m gonna go onto this bandwagon of agile because I know that will force the good practices of DevOps. 

Shane: I have a problem with ops being added to everything just to vendor wash it. But if we took the frame that there is an agile way of working, and when you do that, you can choose some scrum stuff, some lean stuff, some Kanban stuff, and then, you can grab some technical practices from the DevOps community because, continuous integration, continuous deployment, automated testing, they are things you probably want to do to make your life better. As long as you take some of the agile practices and some of the technical practices and mash them up, you are mashing up your own. X Ops, 

Mirco: The key to that is, that it enhances the feedback loop and you’re actually using that feedback to improve. 

Murray: This is all very much what XP was for me when I first started doing it.

Mirco: I’ve missed the XP boat. That lived in parallel to me and I was not aware of it. And we used to call that development architecture. Scrum, doesn’t really have a recognition that problems, incidents, defects will hit the same team. That it’s an actual DevOps team that get hit by these things. And ops is very often very Kanbany because it’s, the highest priority ticket gets taken first. So how do you mesh these two things up? And how do we find mechanisms that allow teams to scale up and down how much Kanban they have, how much scrum they have, so that they understand the difference between product development and fixing problems?

Shane: So when people ask me what DevOps and data ops and all that is it’s a change in behavior where the team now build it, deploy it and fix it. There is no throwing over the wall. It’s coming back on you when it doesn’t work. It comes back on you when there’s a change. So automate the snot out of what do you do because otherwise you’ve got a problem coming. But the other side is as the team build more and more, they are going to start spending time fixing and changing the asset they have built. So you’re going to lose velocity because they are reworking product delivered and you just have to suck it up. 

If actually they’re getting a snot load of changes from the organization because they want what they’ve delivered to be changed, then there’s going to be lots of rework, but that’s okay. If they spend 80 percent of their time reworking for change, and they’ve automated their practices, then that’s the call of the leadership team on where the value lies.

Murray: Yeah, I would agree. And then I would say the only time there is a team that’s called a DevOps team would be, a highly specialized team supporting the other teams, to enable them. 

Mirco: I’m not sure whether you guys have been exposed to the team topologies book. The way they describe is that if you take DevOps team as the kind of a platform team, like an internal service provider, then that makes sense. If you’re at any serious scale, you would get to the point that it’s impossible for teams to have full end to end ownership.

And I say this because the people who say they have end to end ownership, I bet you they’re using at least a cloud, which means they have at least a team behind them, call it Amazon, call it Microsoft, call it Google, that develops a whole bunch of their infrastructure, a whole bunch of their tooling, a whole bunch of the architecture components, right. Now in big, big corp, you will have your own teams that do that. But the core thinking there is that they’re developing a platform and having a product that someone else consumes. And it’s not a, I’m setting the standard by giving you specific tools that you have to use, Mr. Developer. No, I’m enabling you to focus on building the mobile system, the whatever system. 

Shane: I agree. And , when you scale having another group of people that do some repetitive tasks in an automated way for you is good. But what I see is they have a team of 10 people, they rebrand it with the DevOps thing. And so they don’t get the benefit of changing the way they work because they’ve just rebadged everybody.

Mirco: The interesting challenge is there is no one right answer. We know the behavior we want to see at the end. We want people to have ownership. We want people to have visibility and transparency of what they’re achieving. We want the teams not to be hindered by, extra paperwork. But there’s not one answer to get there. And so we all agree on the outcome and then yet still every organization has to find their own way there. 

Murray: We’ve had a number of podcasts now where we talked about how the key to it is to grow your own version of Agile. And it would apply to DevOps too. People coming in with a playbook and implementing it probably is not going to work for you because it’s copy paste from somebody else. It’s full of solutions for their particular problems, not yours, and you just implement it, it may not work for you. So, we strongly encourage people to try things, if it works, do more of it. 

Shane: We also talked, though, about bringing in people with expertise and experience can help you accelerate the way you work. So yeah, guidance and choices and learnings from other organizations. They help shortlist some of the options you might want to try. So while it’s not a blueprint, it’s not an out of the box solution. There are things that other people have learned that are useful practices we could adopt that may help us get there faster. So it’s a balance. 

Mirco: Someone with experience who has done it a few times can help you choose the right tools to try, and even that is not a guarantee that’s the right tool right now, but you have a better chance than if you just randomly pick a tool. I think that’s the best way to see that 

Shane: you see a lot of DevOps teams being mainly Kanban and lean and flow based than scrum based? 

Mirco: I would guess that the vast majority of the DevOps platform teams, that is predominantly Kanban. And it’s for the reason that this is crucial architecture. What that means is that if there’s a problem in production, I’m relying on this architecture to deliver this. So that means you’re now, doing heart surgery when you change your deployment mechanism. And many organizations don’t appreciate that. They still treat these as the IT tools of old that are sitting on the sidelines somewhere.

Shane: So, people often underestimate the amount of investment to put the automation in place. An example I’ll use is to create the templates that we use to deploy services take some time and effort. And often when you’re trying to be agile and you’re trying to do things fast, especially at the beginning, there’s a real habit of, I will just deploy that manually, especially cloud providers. It’s easy for us to go into the console and go clickety click. And we have a service and then clickety click. We have another service and clickety click. They’re connected. And a week later, everybody’s starting to develop, but when we have to replicate that environment or replicate that service, we forgot what we clickety clickety. And yes, we can go and look at it but we can’t deploy it again. And we can’t change it and deploy it multiple times. That memory’s gone. So do you find that the investment to do this well is quite high and often people will cheat?

Mirco: At the broadest level, It’s really difficult when people go into a serious investment in DevOps to see immediate benefit within like a 12 month range. It’s quite tricky at an enterprise level to get that because there’s a lot that needs to come together before you see benefits. So what you’re describing up front to clickety click something, it’s in my view, completely okay if you’re testing that bit of architecture is going to do what we need to do. Investing in the automation and the infrastructure’s code pattern for something that might not work for us might be a waste of time. 

This is an experiment. We’re just going to go quick into the Azure portal, click ourselves a couple of servers together, run this piece of code to see that this works, and then we’re going to stop, pause, build the automation, and then come back to this. And everyone’s going to push us at that point to say no, I need more features. I need more features. Can you just put one more thing into the server? And now we consciously said, we’re going to do this for two sprints and then we’re going to stop and we’re going to take a sprint and build the automation and then we’re going to accelerate again. But that conscious decision making is what requires leadership. That requires the executives to have the backbone to stick to these conscious decisions. 

Murray: What are the common mistakes people are making when they’re implementing DevOps? 

Mirco: I will keep this to three. The absolute biggest one is that people don’t know whether they’re making progress or not. You go in there and they’re investing in DevOps. Okay, so how will you know that you’re getting better at this? And they will tell you things like, we have more deployments or more of our teams are using our standard continuous delivery pipeline or whatever. And none of that means something. How do the people that pay for you actually know that they’re getting value from this investment? It takes a bit more effort up front, but that way you know that these things are persistent. And not just fades where when a new CIO comes in and is not as interested in pouring money into DevOpsness. So being able to measure progress is one. 

The second one that I see is newer, and I’ve seen this only really in the last few years, which is where people want to automate everything upfront or they believe that automation will solve every problem. And the reality is if we have a process at a hundred steps, it’s gonna be incredibly expensive to automate a hundred of them.

But automating the 60 most risky ones, is easy to do and then to optimize the other 40, that it’s easy to get them right. So that a human can interface with that system and bridge the gap. By being able to look at the end to end process and say, Okay, we’re going to take these three bits here and automate them. And these five bits stay manual. And that’s the right way for us to progress. It’s very difficult for organization. Most of them say, Just one more step, Mirco. One more bit of automation.

And then the third one, and I still see that, is the belief that you can just reorg yourself. You just, rebrand some people, put some teams together, and that’s going to make a difference. I honestly believe no one has ever reorganized themselves out of a problem. 

Shane: So we hear the word agile transformation a lot. I don’t think I ever hear DevOps transformation. Is that a term that’s used a lot? 

Mirco: They do use that. There’s plenty of organizations that are doing this and don’t get the benefit from it. But when someone says I want to do DevOps, and I ask, so why is it, and they can’t explain why they want to do it, It just doesn’t work. 

Murray: I, Don’t think I can let you go without talking about big professional services firms. What I’ve seen a lot of is partners and managing directors going in and selling agile and DevOps. Then they go in and they do waterfall, but they just do it in sprints. And then 18 months later the code gets deployed into production full of millions of bugs everywhere. And I, don’t know why that is. But I’ve started to think lately that, maybe all that they’re doing is giving the executives what they want. Which is the appearance of Agile and DevOps without actually changing anything. Do, first of all, you agree with me that a lot of that is happening? And then secondly, Why is it really happening?

Mirco: I certainly see it and I see it from ourselves and others. But the first thing is that as an industry, there has been a trend of carving things up and a level of distrust between organizations. And what that meant is I’m going to give dev to one organization and test to another because I can’t trust my dev partner. So I’m going to give tests to another organization and then I give the ops piece to yet another organization because they will then control the other two and as a client, I don’t have to get involved then because these three will fight it out and then I get the best. result. 

The DevOps surveys, the DORA reports that have come out very clearly highlight that organizations that have functional outsourcing are significantly worse to the ones that have end to end outsourcing. That’s what I see everywhere. And that’s what I fight. And I have been in organizations where I build automation. And then I would ask the ops team to leverage the same automation. And they say they’re not going to do it because it’s not their corporate tool. And you see this in all kinds of permutations.

And so to me, there’s a way of thinking about this. That is crucial of how do we actually have the same incentives. So if you get that one right, I think you have a better chance. 

You made a really good point about, but do they actually want what they’re asking for? I want Agile, but I also don’t want to be involved and here’s 6, 000 requirements and I need a fixed price and you have 18 months to deliver it. And I’m like, wait, sorry, you mentioned Agile. What exactly about what you’ve just given me has anything to do with Agile? You know, like Scrum, Scrum Master standups. I mean, Yeah, cool, you can get those. But you can’t get agile. You can get all the content you want. If you don’t have the core of being able to deliver quicker iterations and not incremental, not just build, you know, feature, feature, feature, feature, but actually improving a solution over time, you can’t get it.

But let’s be also clear, providers need to make money. So if the client wants to buy something that is waterfall, but uses standups. Then there will be people to sell it.

Murray: I proposed to a large insurance company that had divided up their work between all the four different professional services firms and some Indian firms as well, that the way forward was to have a series of product teams. And each team would go through the entire life cycle from beginning to end from idea through to ops. And it would have all the people from all the vendors and the staff in that team that needed it.

So there’d be, some people from the client, some people from consulting company one, some people from consulting company two, some people from consulting company three, and they’d all be in the same team all the way along, taking these features right through seeing if they worked and getting feedback and looking after them.

I proposed that and the managing director of the big consulting firm that was working with us at the time undermined that whole concept because he found that threatening to his revenue. He wanted to either own everything or just to have really clear stuff he owned and then, he could blame other people for what they did. 

Mirco: I’ve changed my mind a couple of times on this. In the early days I did quite a few projects. with joint vendor teams, and I always found it working fine. There was problems along the way, But it wasn’t too bad. But what I come to realize is there’s so much about culture and so much of a line incentives. That is just really hard to find these people. I think I might have been lucky that I had really good people from these different vendors working together, but if everyone is in there to increase their revenue. Then how can their prime target be delivering the best outcome for that client. 

Murray: Whether you can mix consultants from different organizations together or not in a single team depends on the way you structure the engagements. So if you have fixed scope engagements where you’re trying to make vendor A responsible for something, and then you’re going to sue them if they don’t. And B’s got something else and C’s something else, and they’re all fighting with each other over revenue. Then it’s going to be hard for them to cooperate because the executives in the different consulting firms are going to prevent you from working together. 

But on the other hand, if you say to all these different vendors up front, we as the client are taking responsibility. And we want to share it with you by getting the best people from each vendor, depending on their skills. And they’re going to work in combined product teams for long periods of time. It’s a shared responsibility we’re all trying to get the best outcome together. I think that set of incentives would work. 

Shane: Organizations move to a systems integrator model to outsource responsibility of integrating hard stuff. And it’s been an epic failure. Because you can’t outsource that responsibility, at the end of the day, it’s your business that suffers when it’s not delivered. For me, the large consulting group failure is about leadership. It’s about leadership in those consulting companies and leadership at the customer. In New Zealand we have a very small talent pool and it’s got even smaller because the large consulting firms can’t fly in people from their hubs.

So we often see the people on the ground from those consulting companies change company and then they come back on site. But they are great people. They want to get stuff done. They want to be part of a team. They want to deliver value. And so when they’re dedicated to the customer for a period of time, and they’re embedded in the team as a team member, with all the same access and all the same rights and all the ability to have an opinion as well as the people that are permanents with the customer, then the team rocks it. But as soon as we do things saying you can’t have one of our devices or, you can’t be here full time because I’m going to sell you to three other customers. Those are the behaviors that break an agile way of working. And big consulting companies tend to do it more then little consulting companies.

Mirco: Yeah, and I think you described it well because it’s from both sides. You come in and you don’t get the same say and my people go home at 4. 30 and we expect you to work until midnight. And then on the flip side, you share people across multiple engagements.

There’s a promotion cycle in my organization and certain criteria that we use to promote people and I need to look out for those people on client engagement and explain, Hey, this guy needs this kind of next step in his career and hence that’s why I’m moving him into a different role. You can say no to this Mr. Client but then that person will choose to go somewhere else. It’s that real honesty with each other to say, here’s my incentive. Here’s your incentive. How can we get this aligned and where are the cultural barriers or contractual barriers? That’s adult conversation, but it’s not the default in the industry by no means. 

Murray: Have we achieved the goals that DevOps was trying to achieve? 

Mirco: I’ve always the goal that my team shouldn’t exist. At some stage, the goodness will be distributed so wide that I have nothing else to add. So right now there’s a DevOps and Agile peak in my team and there’s lots of stuff. Lots of other people have Agile people and DevOps people and that’s good. At some stage, we don’t need people that are specialized in this anymore. I want to actually build cool shit. I want to do a cool project where I can build a new billing solution or the next banking platform or the digital passenger card or whatever, right? I want to build cool shit, but I’m so frustrated that everyone is building their solutions still in a pretty bad way that I still want to help more of them. When I don’t need to do that anymore. And then I will build individual solutions myself.

Shane: Same thing with agile coaches, we’re only there because people need help. At some stage, I hope we get agile natives where they don’t need coaching because they work in an agile way because they are.

Mirco: So hopefully at some stage they are no agile in DevOps conferences anymore, because we’re just all doing it and we find the next big thing.

Murray: I think we’re reaching the end of our time. So we should summarize. I , really like the way that Gene Kim talks about the way, DevOps works in the Phoenix project. That makes a great deal of sense to me. And, I see that as being very much part of what agile should be about. I think we need a much better emphasis on technical agility. And I think that’s what DevOps offers. Dev ops should really just be a platform to enable the construction of cool things more efficiently and effectively, and to empower the development teams to do as much as possible. And the tools are really not that important. It’s not about the tools, it’s about building things that people value, and getting it out there, and in particular, getting feedback really quickly from the market and from users and feeding that straight back into the product. And if we have dev and ops in the same team with product. Then we have the capability to do that. 

Shane: I’ve got some interesting takeaways. So I think the first one is if I took the word DevOps out of your language and listen to what you said, you’re describing Agile. Some Ops groups are more Kanban or lean and some naturally go to these technical practices and some don’t. The success criteria is, is this feature being used and has this feature added value? So it’s not mandatory metrics of how much we pushed. It’s is what we pushed being used and has it actually moved or changed or made somebody’s life better in terms of a customer. 

The other one is differentiate between experimentation and investment in automation. But when you do invest, differentiate the things that are worth investing to automate and the things you can leave manual because the cost of automating them is just too much the value is not there if you automate it so you still have to do that. 

Murray: Now, how can people contact you, read about your thoughts?

Mirco: So, I’m , on Twitter, at Mirco Hering. I have a blog called Not a Factory Anymore, which describes , my battle against the mechanistic way of thinking about IT and then, hopefully we’re getting out of lockdowns and into conferences. So you will see me there and tap me on the shoulder.

Murray: Yeah. And people can get your book, which is available on amazon, 

Mirco: Yes, DevOps for the modern enterprise. 

Shane: Excellent. That was a great conversation. And we didn’t even talk about Docker versus Kubernetes. So I’ll catch you later.

Mirco: Thanks guys.

NN – Outro: That was the No Nonsense Agile Podcast from Murray That’s evolve with a zero.