Join Murray Robinson and Shane Gibson in a conversation with Bryan Finster about Continuous Delivery and SAFe. Continuous delivery improves quality, productivity, value, and developers’ lives. It reduces the cost of change so we can experiment and learn rapidly.  Focus on improving your process so that you can deploy working solutions every day. Ship one very small feature and see if it has value. Start with the minimum viable Continuous Delivery activities. Solving these activities is the improvement journey.  Never bypass the deployment pipeline. If the process doesn’t work, fix the process.  Organise your teams around domain boundaries. Run CD Dojos. SAFE is bad because everything is tightly coupled, which means you’re as fast as your slowest team. A 90-day plan is a bad plan.  We don’t need a release train engineer when we have a CD pipeline.

Read along you will

Shane: Welcome to the No Nonsense Agile podcast. I’m Shane Gibson. 

Murray: And I’m Murray Robinson. 

Bryan: And I’m Bryan Finster 

Murray: Hi Bryan. Thanks for coming on today. So we wanted to talk to you about continuous delivery and have a bit of a chat about your experience with Safe. Let’s start by getting you to explain to the audience who you are and what your background and experiences. 

Bryan: So I’ve been a software engineer for around 28 years most of the time doing supply chain software. For the past five or six years I’ve been focusing platform development to make it easier for developers to do their job. I founded and I’m the former lead of the Walmart DevOps DOJO.

I’m currently a value stream architect for defense unicorns working with Platform one in the US Air Force. And I’m a passionate advocate for the quality of life improvements that come from actually doing real continuous delivery at the team level. I want every team to live a better life. I also have a, blog, the Five Minute DevOps blog on Medium and I coauthor of the modern cybersecurity book that was released last year. I have a chapter in that book where I give my thoughts on how continuous delivery contributes to secure software.

Murray: So what is the state of software development and continuous delivery in most organizations today. 

Bryan: It’s pretty bad. The reality is, a lot of people think continuous delivery is the robots. I have Jenkins, We’re doing continuous delivery, and it’s a much more disciplined approach to software development than I think most organizations really understand. They’re struggling with the basics, the fundamentals of continuous integration, trunk based development, breaking things down small, and so they’re not really doing continuous delivery.

 Last year at DevOps Enterprise Summit, several of us came together and said, We really need to help ’em. And we put together a website, minimum cd.org, where we have the minimum viable, continuous delivery. So you can say, yes, we’ve started on the path to CD. We put a lot of work into it, a lot of thought into it, par it down to the very bare minimum. Dave Farley agrees he signed onto it. So we must be on the right track there.

Murray: What is the minimum necessary for continuous delivery? 

Bryan: So number one, you need to be doing continuous integration which means that assuming we’re coding today, our code tested working to the best of our knowledge code lands on trunk today so that it can be fully vetted with everybody else’s code. That’s real ci. CI is not a tool. 

And then the application pipeline is the only way to deploy to any environment, which means that if we have an emergency change, we have to make, the only way we can make that emergency change is with the application pipeline. We can’t bypass the pipeline. 

The pipeline decides whether or not the change is releasable. We don’t have an approval process outside the pipeline. The pipeline’s verdict is definitive. It’s absolutely deterministic. That every artifact generated by the pipeline meets the organization’s definition of deployable. And that’s different by organization, but you define what deployable means to you, and then the pipeline validates that. 

The pipeline generates immutable artifacts. We don’t modify an artifact after we push it into get. If it fails, it’s done. If we deploy it, we can’t change it. Don’t use snapshots. All feature work stops when the pipeline is red. And this is a behavior thing. If the pipeline breaks, our highest priority is the ability to deploy to production. Every change you make on top of it is building on top of quick send. So we need to fix the pipeline and then go back to future work. 

We need production like test environments. That’s super important. If I’m testing under one configuration and deploying another configuration, how much confidence do I really have?

We need to be able to roll back on demand because we need to be able to fix things rapidly. What’s one of the reasons for continuous delivery and application configuration deploys with the artifact. We version these things together. That’s the very bare minimum. 

We actually paired this down. People had lots of good ideas of things that should go in there. And we said look, number one, that’s not true in every delivery context or it’s not absolutely required. It’s an excellent idea. Let’s add it to the beyond the minimum section. So that’s what we did. We worked on this really hard for several days with a big group of people and we now have contributors from all over the world. We’ve got 38 contributors globally, and 120 people have signed on it and said, Yeah, I agree. And if wants to contribute, all they gotta do is submit a poll request and start a conversation.

Shane: So what is the difference between continuous delivery and continuous integration and DevOps, or are they really just all one and the same thing.

Bryan: No, they’re not all on the same thing. CI has been around for, over 25 years. It predates xp. It was just written down in the extreme programming book. So continuous integration is the activity of everybody integrating their code together onto the trunk at least once a day minimum. And then having tests run against those to validate that all the code runs together. CI is all of the activity that happens pre-deploy. So all the tests we run, pre-deploy, static code analysis, the unit tests all those. Once you have an artifact that’s versioned and then start deploying it to test environments, that’s where you have the acceptance tests that run in the rest of the CD pipeline. So that’s how I segregate those two in my head. 

DevOps that’s one of those terms that is so widely misused. I always used Donovan Brown’s definition. DevOps is the Union of people, process and product to enable the continuous delivery of value to our end users. DevOps is the entire way the organization operates to enable the continuous delivery of value to the end users. So CD only really works in a DevOps environment. You can do CD type things, but you’re gonna get blocked up and down the value stream. 

Shane: Yeah. It’s, around ways of working as much as it is around technology. It’s around processes and patents and practices and team topologies

Bryan: It’s, tearing down the silos, architecting things well. I come from a supply chain background, and I think of this as the supply chain of communications. 

Murray: Let’s say I am a head of product in an organization. Why would I wanna do it? It sounds like a lot of work and a lot of money and a lot of effort. I just want to ship features out to customers. What’s the value to me of doing continuous deliver? 

Bryan: Number one, so that I can ship value to my customers. I’ve been a developer for a long time, and most of the time doing it wrong and being woken up in the middle of the night fixing my stuff that’s breaking in production. Continuous delivery standardizes our release process, which means that I have a standardized way to validate the quality of my change. Which means that at three o’clock in the morning when I’m super tired, I’m not forgetting steps to validate a change before it releases production. And I’m not making a dumpster fire at three o’clock in the morning. 

It also means that because I have the standardized process and I don’t have to depend on memory to get it done, I can fix production faster and go to sleep sooner. I think that’s super important. It lowers the cost of every single change, which means I can make smaller changes. Every line of code we type is a potential defect. So how many lines of code do you want to shove into production at the same time to see explode With cd I can lower the cost to change to the point where I can afford to deploy trivial change and verify that it works fine.

 CD’s also a forcing function for improving your value stream. It’s a forcing function for identifying where you’re spending money on toil that you don’t need to spend money on and improving the efficiency of the flow of information through your organization. When you start going why can’t I deliver to production? And you start hammering away at those things, you start at the team level and you find out, Oh, there’s these team issues. We’re not breaking work down small enough. We don’t test well enough. But eventually you get those cleared and you find out, we have all of these constraints in the organization where the organizations just architected incorrectly or has all these processes that are scabs from poor quality in the past that we need to fix to be able to deliver.

And one more that speaks to me closely because this is why we are pushing for CD for all teams in Walmart, was that you have to level up as an engineer to do continuous delivery. And when you have this challenge put in front of you that, hey, we need to deliver working solutions to the end user, not code, not change, working solutions to the end user daily as a team. You’ve got a lot of problems to solve in solving those problems elevates you as a team. It elevates individuals on the team as engineers, and that’s not theoretical. I’ve witnessed it. I’ve lived it myself. And that’s why we were pushing for CD at Walmart wasn’t to show change out faster. It was to elevate engineering excellence across your enterprise.

Shane: I think that’s a good point you raise. You often hear that Amazon are shipping hundreds of changes a minute, is that the concept behind it, 

Bryan: Yeah, no, absolutely. If I can release one line of code and verify it’s good, man. I feel so much better. When you’re releasing every quarter, who cares? I’m spending all my day coding stuff and I never get to see it work. But it’s fun to release and see your stuff working.

Murray: Now, let’s say I wanna restructure something in a substantial way, like my user interface design. I don’t want to release a small part of it because it’s actually not gonna work with the rest of it.

Bryan: Yeah, 

Murray: so what am I doing? Am I, releasing changes that are in progress into production, but they just shielded. Usage, Is that what I’m doing? 

Bryan: Absolutely. Number one, if we have to make a huge user interface change, boy we did something wrong. But let’s just assume that that’s, true, right then. Yes. You separate out release from Deploy, so we’re deploying small changes all the time. I have an environment where I have some way to segregate off a group of people who can see the changes as they’re happening so that we can get feedback on those things. But majority of the users or all of production doesn’t see that change until we decide to activate that change. The goal is feedback. We don’t wanna code something in a complete feature on my laptop and then deploy it and find out it was actually the wrong thing to do at all anyway. We need the closest to production feedback loop as we can possibly get. And so you’re releasing all these small changes, but you’re absolutely segregating it off from the end user. When your marketing’s ready, when your documentation’s ready, when all these other things ready, then you make another change to activate it, and then if it blows up, you just make another change to deactivate.

Murray: Sometimes you do need to make big changes, because otherwise what you’re doing is a whole lot of local optimization. You can’t just do it through one line of code at a time cause it’ll break everything else. 

Bryan: You can make it through one line of code at a time. It’s just a matter of what kind of change you’re trying to make. There’s different techniques, feature flags are a technique. I could replace an entire service as a change, but what I would do is I would deploy a new service as version two, not change the current service.

I would make sure I had parody on the things that mattered, and I would make sure that the new things were working the way I needed to before I would switch the load over to the new service. An example that I often give is that we were given this, requirement for printing labels for distribution center. And depending on the kind of freight and where it was going, you had, we were told there was four different kinds of labels, and so we had some simple rules and it was basically a switch statement. Well, it turns out there was actually 34 different kinds of labels and they just didn’t let us know and they were feeding ’em to us a little at a time. And so the switch statements started getting bigger and bigger until it was just, Are you insane? There’s no possible way to make this thing work. I went out and implemented it an entire rules engine for label printing. And it took two or three weeks to implement it. But I was pushing change to production daily. It’s just the last change I made switched it from that, that switch statement over to the rules engine.

Murray: Yeah. 

Bryan: Yeah. You just have to engineer for release. That’s what you’re doing. It’s just engineering. the key thing there is that it’s not about that. It’s about the ability repair production fast while you have things in progress without having something on its long lived branch that could possibly get truncated because of other change that makes it incompatible. You need all these things happening at the same time. It’s just an engineering problem.

Murray: Yeah, perhaps the other benefit I would see as a product manager is that I can test things much more easily. Not technical testing, but market testing and customer testing. 

Bryan: It lowers the cost of change. And by lowering the cost of change, we’ve reduced the cost of running experiments. We don’t have to have a huge plan to run an experiment. When one of the pilot teams we had for CD at Walmart were releasing warehouse management software to the distribution center we had a Slack channel open. And the people in the DC would give us ideas and we’d implement the idea in a couple of days and spin it back around and say, What do you think? And if they liked it, we kept it. If they didn’t, we threw it away because the cost of change is so low.

Murray: Yeah. so there’s some argument over whether continuous delivery is part of Agile or not. I think it is because I see Agile as being more an XP thing, not as a Scrum thing.

Bryan: Yeah.

Murray: What are your thoughts on that? 

Bryan: I’ve got mixed feelings about it Because number one I’m not a fan of the word agile. I think it’s been completely torn apart by the agile industrial complex.

Murray: Yeah. 

Bryan: If you used agile as an adjective, fine, if you used as an adjective, it’s not possible to be agile in 2022 without continuous delivery. If you’re saying we’re agile and you’re not doing cd, you’re lying to yourself. The very first principle of the Agile manifest our highest priority is to satisfy the customer through early and continuous delivery of valuable software. If you’re only releasing changes once a day, you’re slow.

Murray: Yeah I see continuous delivery as an evolution of the technical side of agility.

Bryan: I don’t see it as the technical side of agile. Whenever I draw a diagram of continuous delivery, it starts with an idea cloud that gets shrunk down through a funnel into very small changes. Implemented by the team, it goes through a string of robots, it goes to an end user, and then there’s a feedback loop to the idea cloud. It’s all of everything. It’s not the robots, it’s not the team, It’s the entire product development cycle. And so if you’re gonna say we’re doing agile, then it is agile.

Murray: Yeah. Yeah. I agree with that. I wanted to ask you, how do you transition a team to continuous delivery? I’ll give you an example. I was working with a team a while ago that worked in silos. They had outsourced half of their development to an offshore team, And half of their testing was very separate from Dev.

Bryan: Yeah. 

Murray: And there was a lot of conflict between the managers at the silos and everything was really late and really buggy and way over budget and time as you would expect in an environment like that. They had automated tests, brian, but they turned a lot of them off cause they kept coming up as red. So you can’t have that when you’re deploying. And so they would only update their automated tests every few months when they were gonna do a release. 

Bryan: Let me ask you something very important. Who did support for their product 

Murray: they did their own support. 

Bryan: and they turned their tests off. Were they insane?

Murray: They run under a lot of pressure from management,

Shane: Yeah, but remember you can fix it in production, you just go in and change the code in production as a quick fix to stop that problem that got all the way through that bad coding practice and bad behavior. 

Murray: It was a complex product, they had other teams building things for them that they built on and they all using the same sort of approach. So what would you do? 

Bryan: So the very first experience I had was with Walmart’s main warehouse management system, which was a 25 million line entangled monolith with a shared database with over 700 tables and 800 applications. All with about 400 developers on teams. But they were all working on the same code base. They were basically feature factories. There was no segregation between capabilities. We were delivering every quarter was scheduled war rooms that lasted for a week for 24 7 support, and generally went longer than that. They wanted us to deliver every two weeks. That’s where we started on the path to continuous delivery. So just imagine you’ve got this level of nightmare, and you’re gonna now escalate it to every two weeks. Instead, every quarter

 We rearchitected the teams around business capabilities. That’s step number one. Everybody should study strategic domain-driven design. We did a domain decomposition of the entire warehouse management system. We told management, this is what the teams need to look like for this. We spun up the first set of teams to do that. We defined the interfaces between those teams.

We told those teams they should run as fast as they could, and that they should not be coupled in any way other than through those interfaces. They should be able to deploy in any sequence. And then we started driving down to getting clarity around the work itself and started focusing on how do we test better.

Because of the way we are delivering, we no longer met the SLAs to hand our code off to ops for them to support it. So we had to support it on code, which was another important feedback loop on quality. And then we focused on how do we improve this and get this better and better all the time. That’s a broad overview. I’ll say that most teams, what they’re struggling with is they don’t know how to break work down small. They don’t know how to test very well. And in your case, they get really crappy management needs to be fired.

Murray: Yeah. So there’s a common problem you have where you’ve got developers and testers who are used to working in the silos, and you put them together in one team and they just sit there and say, Where’s my spec? Where’s my test plan? Where’s my test case? Particularly if they’re offshore in a developing country. My job is to execute test cases. Give me a 10 page test case and I’ll execute it. So what do you do there? 

Bryan: I’ve lived this. So the worst possible case is you have a QA team, because then actually why would the QA teams test be correct? They’re using the same requirements you’re using as a development team. They’re just misunderstanding it differently than you are. 

Murray: And often the requirements have changed and the testers are three months behind the developers and what the requirement should be anyway. 

Bryan: Yeah. Yeah. Well, but I’m just talking about in the case where they haven’t changed. The best cases, they’re misunderstanding them differently. On one team I was on, we had offshore testers and once we started taking over testing because we had to, for ci, we started understanding how poorly they were testing things. We went to the QA department and told them, You need to reassign them to another team because we can’t use them anymore because we’re testing better than they are. And here’s the proof.

The way that Dojo’s work is it’s immersive learning. You bring a team in and you teach them how to do the work by working with them to do the work with their work, not on contrived work. In the Dojo is if there was a tester on the team, we would say, You’re now a quality engineer. You need to stop testing. You need to help all of these developers learn how to test better, because there’s no possible way that you can handle the throughput of all the code that’s gonna come at you when we get these guys on ci.

Shane: So you talked about breaking up the team, the monolith code base into domain boundaries. Can you gimme an example of what the original domain boundaries were for each team?

Bryan: So you have to think of all the activities that happen in a warehouse. So you’re receiving new freight, you’ve got inventory management, you’ve got invoicing, you’ve got shipping, order filling And order processing. So we started forming teams around those business capabilities. Now, in some cases, the business capability was too big for the team. So for example, the team I was focusing on initially was receiving. We had to break up that business capability into inbound order processing, the purchase order stuff, and then the actual act of receiving because it was just too much work. So we split the business domain and defined an interface between those two domain And so it’s teams focused on a business capability and becoming business experts at that capability with interfaces to the teams they touch.

Shane: Yeah, so I’ve always thought about it as, core business processes. At a macro level, if it’s an eCommerce company, it might be customer orders, product, customer returns, product to store store ships, product

Bryan: Yeah. Shopping cart. Yeah. Yeah, exactly. Yeah, 

Shane: And then what you’re saying is, with some organizations, even that’s too big. So we start going down to those tasks to be done. Those admin tasks of invoice processing, invoice payment. 

Bryan: We literally drew boxes on a whiteboard and wrote a sentence for what that sub domain did. And if we need to split it, we say, Okay, What does each one of those sub domains do? What does the single responsibility for each one of these sub domains. What’s the object you’re responsible for.

Shane: So can you explain what the Walmart Dojo was, how did it work and what was the value from that? 

Bryan: In 2014 Ross Clinton and his team at Target started this idea of bringing teams into an actual space and learning how to do their work better using their work. Focusing on improving delivery, not how well do you do standups? Capital One copied them. Verizon copied them. John Deere has one. And there’s actually a Dojo consortium. We all communicate with each other and share ideas. 

Murray: So this sounds a lot like an enabling team in team Topologies.

Bryan: Yeah it’s similar to that. Different dojos operated different ways, but the way the Walmart Dojo operated was that two of us would go and join a team for six weeks. We had a very dedicated process. We copied from Target where you’re gonna run two and a half day sprints every two and a half days. We’re gonna deliver, we’re not, and we’re gonna have a retrospective and demo every two and a half days. We’re gonna learn to break down work so it fits in less than two and a half days, and we’re gonna get it done. Let’s track our metrics on how we are on continuous integration, defect rates throughput, that sort of thing.

Murray: I wanted to come back to offshoring Brian, because it’s very common pattern we see, in corporate it. Why would we pay a thousand dollars a day for somebody like you when we can pay 200 bucks a day for somebody in developing country. Surely you’re all the same

Bryan: Yeah, we are replaceable cogs the machine, aren’t we 

Murray: What’s your experience been? Is it possible to do it successfully?

Bryan: It’s a matter of degrees. How much quality do you want and how much do you wanna spend to get that quality. So if you’ve got something that’s cookie cutter, I honestly don’t understand why you’re developing it. Just buy it. It’s a commodity. But if you have something that’s business critical to you, wouldn’t you want business experts who understand your business and care about it, who also know how to deliver that really well? Or would you rather just throw requirements at somebody who doesn’t understand your business or care about your business? They’re just kind of implementing what they read and sending it to you and saying, Is this right? And then they’re gonna get paid on the back end as well to get it fixed,

Murray: What did Walmart do? Cause they’re notorious for being very tied around costs.

Bryan: They’ve got like nearly 19,000 developers worldwide. There was a point where they had a bunch of offshore development, and then eventually they just opened an office in India. And of course that was a big culture shift for a lot of people. But what we really pushed for and what we aimed for was that the teams in India actually owned something. We didn’t just ship them features. They owned capabilities,

Murray: Yeah. 

Bryan: They were Walmart teams. They’re not just feature factories, coding pools out there that we throw code at. I had contractors on my team, but they were team members. It’s just that they were staff augment. They weren’t full badged. I had to go fight for them for things that were ridiculous. I had to fight for them so that I could treat them like team members, but they were team members or we were not effective as a team. You optimize for delivery of value. You don’t optimize for cost.

Murray: Yeah, absolutely. Yeah, my approach has been to treat people as one team. I call it a one team concept. Whether they’re from Accenture or Deloitte or India or UK, or America, let’s treat them all as being human beings in one team, delivering 

Bryan: I agree. We gave a talk about the impact of DevOps on a team at DevOps Enterprise in 2017, and somebody asked what the percentage of contractors were, and I said, It doesn’t matter. What you need to do is you either treat them like team members or get rid of them.

Murray: What’s the impact of continuous delivery on the life of a developer? 

Bryan: Oh man, you’ve never seen so many people who just enjoyed coming to work. Just the amount of teamwork it takes to do it brings the team closer together. But then also just the endorphin hits you get from delivering and getting feedback and seeing your work actually being valued. The morale improvement is amazing. A warning to management, don’t send your teams down the path for continuous delivery and then get cold feet, because they’ll keep driving towards CD and once they get a taste of it, but maybe not for you, so you’ll only just make the high performers leave.

Murray: Yeah. 

Bryan: I’ve seen multiple situations like that. That’s one of the reasons my wife and I are giving this talk about the rise and fall of DevOps, on October 18th,

Murray: Brian, you wrote a very interesting article about your experience with safe, this scaled agile framework for enterprises, and I believe you’ve had quite a lot of hands on experience with a safe environment before you move them to continuous delivery. So what’s the issue with safe?

Bryan: I lived for three or four years under a safe regime. It didn’t make anything better. When you’re trying to deliver better, coupling teams together is the opposite of things that you want to do. Having two week waterfall sprints is not what you wanna do. Having a project plan for everything we’re gonna deliver for the quarter. That’s not agile, that’s a project plan and it’s a bad project plan because you’re not actually going out there and finding out what your critical path is. You’re not going out there and doing any of the real discipline around project management.

 Safe doesn’t work for software development, it’s just the wrong tool. It doesn’t help anybody deliver better. It’s basically making people feel good about agile by taking agile words, layering it on top of a project management process, adding more roles for more people to get paid for things like release train engineer. That’s what a pipeline is. I don’t need to release train engineer. I hit deploy with my finger. And trains how agile is a train. Even containerships are more agile than a train, and then tying all these teams together to you’re as fast as your slowest team. Why don’t you just solve the engineering problems, actually solve the problems that layering on these processes that cause people pain and just suck the life out of everybody.

 We threw it away and found out that everything got better. In the Dojo, we called ourselves safe crackers because we go into teams that we’re currently using safe because their leadership wanted better delivery. And so we had said, Look, hey, you want better delivery? Cool, all this other stuff, it’s killing you. Here’s exactly why it’s killing you, and here’s how this will make it better. Let us make it better and we’d burn it to the ground because their leadership wanted better delivery. 

Murray: so why is safe killing you though? Is it tight cup leg and tight dependencies, 

Bryan: One of my favorites is feature teams. You know what a feature team does? A feature team donates features to other people generally without their permission. They don’t own anything. They don’t own the outcomes at all. They just push features at people. Intersourcing. You have to have a working agreement as permission and submit a poor request. And like you understand that people are pushing these things to you and there’s like a real agreement, a feature team just throws code at you. They’re a feature factory. It’s like you own a capability. There’s a bunch of stuff in the backlog in these teams. Let’s just go clear those. We’ll code those and throw ’em at the teams and go onto the next.

Murray: So why is safe so popular with management then 

Bryan: It’s excellent marketing. There’s a website I’ve put together that is a parody of scaling frameworks. The tagline of the website is enterprise transformation without the risk of culture change. That’s why safe is so popular. You don’t have to change anything. It’s just relabeling things.

 Companies don’t implement this stuff because they hate people. They implement this stuff because they’re sold it. I’ve talked to several safe trainers who will say what you experienced, that was just the first step. It’s supposed to move beyond that. But here’s the thing, enterprises that are big enough that they think safe is their solution. By the time they train everybody on the entry level, it’s too expensive to retrain them for the next step. So the entry level just becomes the thing we do because we’ve all trained on it.

Murray: Yeah. I’ve heard safe consultants and trainers say it’s a really good starting point. And then you’re supposed to change. You’re supposed to evolve through your retrospectives, but nobody ever does, as you say, because there’s no additional training for how to evolve to something else. They will just keep updating the safe from 4.5 to 4.6 to 5.1 to 5.2, whatever it is, and then you’ve, you just implement the next version of it or the updates to it. So it’s like an operating system is the way it’s defined. You don’t get to change it. 

Bryan: The other thing that I’m told frequently by apologies, is that there’s lots of good ideas in there and I agree, and I know where they crib those 

Murray: Yeah. They stole them from everyone else. 

Bryan: Yeah I’m friends with lots of those people. I’m gonna go see ’em at DevOps Enterprise Summit.

Murray: Yeah. 

Bryan: But they’ve taken the good ideas and twisted them to be bad. You have to know the difference between the two. And if you know the difference you don’t need safe. And if you don’t know the difference between the two, you’re left with a cookbook. It’s got lots of good recipes in it, but some of the recipes will kill you. Good luck.

Murray: I think program increments and release trains are the two really bad ideas that should be got rid of. If it was just a patent library with some advice on how to pick different patterns and tailor for your context, I wouldn’t mind that disciplined, agile deliveries like that, That seems quite good. But the program increments, which they just say is. Scrum scaled up, always turns into these quarterly fixed scope project plans.

Bryan: Yeah, and they absolutely do. And full disclosure, I’m not a friend of Scrum either. The framework I prefer is the Get shit done framework. What’s the highest priority item and how quickly can we deliver that and get feedback whether or not that was the right thing to deliver,

Murray: I quite like Scrum, but I think it has some limitations that become very apparent when you scale it up to three month sprints. 

Bryan: My advice to anybody using Scrum is use one week sprints.

Shane: With data teams, I typically recommend three week sprints until they can get a release into production and then bring it down, so I find that they struggle to do it in two, and we get lots of double, triple sprints for a release. 

Bryan: So let me gimme another perspective on that. Let’s go back to the Dojo. We walk into, teams are doing two week sprints. We say now we’re doing two and a half day sprint. Failure’s an option. But what we’re gonna do when we fail to deliver is we’re gonna have a retrospective and say, Okay, now what were the things that were preventing us from delivering? How do we fix those? And then we do it again. We fail again. And those teams would generally fail for about three weeks.

Shane: So, did you have a whole lot of patterns and practices already in place for them to leverage? Or were you getting them to start from ground zero every time? 

Bryan: We had some good practices around how to decompose work. We had practices we had used cuz we were all senior engineers and agile coaches. We were all people who’d done cd and we knew using behavior driven development to slice work down small was super effective, so we knew those sorts of things. We eventually codified them so they could read them and stop bothering us. But teaching teams how to slice work in the tiny increments and then in two, three weeks they’d be able to deliver over two and a half days.

If you’re struggling, deliver in three weeks, drop it to one. I would always start at a shorter sprint because it’s okay to fail, but if you spend three weeks on it, it’s gonna fill three weeks. You’re gonna struggle to get it down to two. But if you start really small, then eventually you can make it bigger. So start at two and a half days, go to a week and you’ll be really happy. 

But yeah, I agree about the release trains. It’s terrible, that’s chaining teams together. We decided to fix this with all teams must be able to release independently. Any sequence at any moment because otherwise, if you’re fixing production or you gonna put the release train together to fix production, or are you just gonna hope you didn’t break anything because of all the coupling?

Murray: yeah. Safe accepts all of the dependencies that exist in the organization pretty much. And then just tries to manage them with red string and all its giant planning sessions. 

Bryan: Yeah, you don’t manage dependencies, you handle em with engineering. And the fact of the matter is that safe assumes that all you’re doing is delivering features, but the reality is that a big chunk of our time is fixing things that break and we have to be able to do that safely. Feature deliveries how we verify that we can fix production rapidly. You don’t optimize for future delivery at the expense of being able to fix production safe.

Shane: So there’s a big buzz in the data world at the moment around a thing called a data contract. Really what it’s saying is in the software engineering world, there’s a a programmatic contract of what an API looks like, and you don’t break the contract. Yeah. You typically version your contract. So you can always know where, where a safe aPI is.. 

Bryan: You don’t have to version it. It’s just like a database, I can add columns all day long and it doesn’t break your queries. But if I go and rename a column, move a column, delete a column, that’s a breaking contract change. And so I’m surprised this is actually a new conversation in the data community because this was fundamental to what we were doing back nearly 10 years ago when we’re talking about how do we decouple these teams? We established contracts between them. We established contract tests between the teams. Virtual services between the teams for testing to verify that we were meeting our contracts, both provider and consumer. And then we were always doing evolutionary contract change. We weren’t making breaking contract change. I know how to version a contract. I’ve never actually had to do it because I’m always evolving the contract in a way that’s non breaking. 

Shane: Yeah, so in the data world we lightly couple our software engineering and our data teams, so the engineering teams are focused on getting the applications built. They don’t care about the data structures. As long as it meets their needs, they’re happy. And we get this data exhaust that the data team has to pick up and reuse. And of course it gets changed. Often the data teams aren’t going in through the production APIs to get the data. They’re using change data, capture technology, and going into the database directly because it’s easier for them. And so now we have a unofficial contract, which is bound to the structure and the database that you are using, but you don’t know that. You don’t know what I’m using it for. We have no contract in place. So as soon as you change something that’s okay for you, we get a breaking change in our data feed because that technology is so closely coupled and that’s where the concept of data contracts come from. 

Bryan: Here’s the thing. Delivering value requires discipline. You can’t just cowboy it. The example you’re talking about there was very similar to that testing team we had that was offshore, that we told QA that we were testing better than them. Instead of using the applications API to insert data, they were manually inserting data, blowing stuff up, reporting bugs on us, and then we’d have to go stop what we were doing, verify that there was actually a bug, and then say, No, it’s because you didn’t insert the data correctly, because you didn’t use the API that’s doing it correctly. Why would you backdoor things instead of flowing the data the way the data is designed to flow? If you’re backdooring things instead of doing that, then there’s something wrong with the primary flow. Fix it. You should never flow data other than how data should be flowing for the primary flow. That’s just absolutely insane. That’s just people who just want to hurt themselves for some reason. I don’t understand.

Murray: They don’t want to cooperate with the application team cause they think they’re a pain in the arse. 

Bryan: It’s amazing how hard people work to be lazy. They should be lazy like me, I’m super lazy. I don’t wanna wake up at night. I want everything to work the first time. I test my code because I’m too lazy to debug it, 

Murray: I have one last question. You said before Brian, that project management simply doesn’t work for software development. Can you explain why?

Bryan: Absolutely. So I like analogies and currently I’m working with the Department of Defense. I’m gonna use weapons analogies. If I’m ordering a hundred tanks or 10,000 m sixteens or enforce these days. I can absolutely put a project plan together. I know exactly what the thing is that’s being delivered. I know exactly the quantity, I know exactly the steps. I can put a complete project plan together with all of the steps required to get that done. I can judge how far along the project plan we are based off what’s been delivered so far. It’s a hundred percent that’s what it is. 

Software development is product development. We’re not shipping M four s. We’re designing an entirely new weapons system. We know that things need to happen and we know what the outcome should be, but we don’t exactly know how we’re gonna get there. And so every line of code is an experiment and is this the right thing to do towards what our goals are? And how do you project plan that? And that’s why product development is very much different from project management. Project management. We have a known outcome and all we have to do is lay out the steps to get there. With product development, we have a thing we think is the known outcome. We’re verifying every step of the way.

Murray: Yeah, I hear, CTO’s and project managers say, software should be like a factory, I should just be able to crank the handle and features come out the end. But I say to them that software is like designing a new car model. It’s the next Tesla. And also designing the new factory that’s gonna make that new car. Software only becomes a factory when it’s in operation. That’s only time it starts to look like a factory. 

Bryan: Yeah, a hundred percent agree. We’re slowly tweaking the clay model. We’re checking to see if the brake calibers we want to use will fit, or if we should use different brake cas even after a prototype car is built. We’re tweaking that. We don’t know what the actual car is gonna look like until we define the assembly line. And that’s all the robots.

Murray: Even the assembly line might not work very well to start with. Like the Tesla assembly line was terrible when they started. 

Bryan: absolutely. And so that’s why project management works really well for Bridges. It only works for houses if you’re building tracked housing where every house is the same.

Murray: Although, to be fair to project managers, Brian, there is a whole stream of project management, which is about adaptive plans, rolling away planning, continuous delivery. You can actually even look it up in the pmi. Not many people use it though.

 Let’s move on to summaries. Shane, what you got

Shane: Yeah, and another great one. I really enjoyed this. So one of the benefits of CI is quality of life. We enjoy our work more. The teams have more fun. And that’s definitely something that I’ve seen with the agile teams I’ve worked with. 

I like the way you described the difference between C I C D and DevOps. This whole idea that agile DevOps ci, that they’re all ways of working, and we borrow and iterate on patents from all over the place to find that way of working that works for us.

But Also like the fact that, you came up with minimum viable, patterns. Get to this level first and then worry about all the extra things you can do. So I like that.

I love that comment about bypassing the pipeline. You never bypass the pipeline because it’s that’s where the value is, having the machine do that work, we get a whole lot of benefit out of it. It doesn’t involve a human getting in the middle to slow it down or introduce risk. So there’s no hack. The process is the process. If the process doesn’t work, fix the process. Don’t hack outside it.

 I like that idea of reducing cost of change. This idea that if we can make it fast enough, if we can make it agile enough, if we can make it quick enough, if we can make it cheap enough, then we introduce the ability to experiment, the cost of failing is so low, we will keep just trying it. And so we see less bundling, we see this idea that we just ship one very small feature and see if it has value. We don’t have to wait until we have 200 features into a quarterly release because there’s no cost. It’s actually more expensive to do that big release than it is a smaller one. Changing one line of code has a lot less blast radius if it goes wrong than 200 lines of code that are changing many things.

 And I love the idea that scabs that have formed because of poor quality in the past. I thought that was a great analogy to use. 

And you have a comment and if you’re only releasing value once a day, then actually in today’s world, you’re slow, get faster, break it down, make it smaller. I like the idea around domain boundaries. That we are looking at our core business processes to figure out where we can actually break it down time and time again. And we focus on those boundaries and we are really clear when these are merging of boundaries, because that’s our danger area. 

And love the fact that dojo’s a bigger than Walmart. So I’ve got a whole lot of reading now to go and figure out all these other organizations have done it and Hopefully I’ll find a customer one day that will let me experiment with their teams around building a dojo. Cuz it’s just one of those things I think I wanna do. 

I love some of the safe comments. So I’m gonna reuse these soundbites, how agile is a train. You’re a fast is your slow team really? Is that what we wanna aspire to? 90 day plan? Yeah, it’s a plan, but it’s a bad plan. And my favorite release train engineer. Yeah, I can do their job with a finger when everything’s working . So each one of those is highlighting a problem with safe riders and showing you where there are areas that we can improve on without following the safe mantra by the letter of the law. So that was me. Murray, what do you got?

Murray: Yeah, I started with Agile, with Extreme programming. And it really feels to me like continuous delivery is the real modern day evolution of what Agile was supposed to be about. And Dave Foley says this as well, that continuous delivery is an evolution of xP and Agile. I think there’s massive benefits to it. It does require higher level of skill and capability, and when people have low skill and capability, it reveals it and sometimes means that you need to change supplier or bring work back in house. If you’ve got slow skill people doing it for you in another company. And that’s a good thing because the benefits are gonna be really high when I have seen people doing it well and It’s amazing. You go into production and there’s No defects, you start to say, Why would there be any defects? We just don’t allow defects, because we’ve tested everything really well and fully integrated in environments and even if we do get something we can fix it quickly. So I think that’s great. I appreciate you pointing out that continuous delivery is about the whole value stream, not just the technical side of things. I see that. And the discussion about Safe was pretty funny. 

You’ve had three or four years in Safe and you’ve got your Safe Parity website. 

Bryan: It’s not a safe parody, it’s just a parody of scaling frameworks. Yeah. Gotta be careful. There’s two URLs. There’s scaled agile devops.com or if you’re lazy like me, it’s just sad. f.com. 

Murray: I completely agree with your criticisms of safe. People say that if you are already agile, then you can do safe in an agile way and tailor it for your environment. But then I say why would you want.

Bryan: Yeah. You already passed the need. I get a lot of joy out of the SAD F website. A lot of people comment on it or retweeted who know it’s a joke. I’m actually giving a talk at DevOps Netherlands on skilled agile DevOps maturity framework because they think it’s funny.

Murray: Brian, how do I become a certified trainer in the MF framework so I can make lots of money? 

Bryan: So there’s two ways. You can either send me $5 and become a certified practitioner. You send me $20 and you can become a sad fellow. Or if you submit an experience report of how you view scaled agile DevOps principles in your organization, then you can become a Scaled Agile DevOps accredited facilitator. That’s fun and I enjoy it and that’s how I rant some. 

Now, I’d like to leave everybody with this. The thing that helped the teams to really accelerate was not scrum or a release train. It was, why can’t we deliver working solutions today? Just that question every single day, daily retrospective. Why can’t we deliver working solutions today and just hammer away at those problems? That’s the most effective way I know to improve your value stream. 

The minimum CD project we put together, minimum cd.org we’ve gotten a lot of feedback that’s actually helped people, that they’ve actually started using that in organizations to try to drive change. We’ve got information out there like how to start the journey with a lot of the challenges teams go through along the way. The reason for the things we laid out is that these are binary challenges. These are things that we can say, yes, we are doing them, or no, we are not doing them. It’s just a yes no thing. They’re not principles, they’re activities. Solving these activities are the journey to improvement, so you figure out why to solve these things and you get better. 

Murray: And the thing we gotta keep coming back to is the reason we do this. If you’ve done this well, you don’t want to go back cause your life is much better as a developer product owner. 

Bryan: It’s better for the end user, which means it’s better for the organization’s bottom line, and it’s also better for the people within the organization. It makes it better for all of the people involved.

Murray: I think you’re Gonna be much more competitive as a company 

Bryan: Yeah. People have told me, Oh, we don’t need cd. I’m glad you have no competitors in your market space. Insurance is the one that boggles me because insurance is entirely fictional electronic stuff, and you’re not doing continuous deliver. Oh my God, Amazon does, and they’re dealing with physical things, and you’re only dealing with digits. You’re terrifying me. 

But also, I talk a lot about these things because I, really want every developer’s life to be better. That’s why I talk about these things,, if you just go to blog dot brian fenster.com or rant brian fenster.com, either one of those will take you to my Medium blog. 

Murray: And the other website is minimum 

Bryan: Minimum cd.org. And again, that was not me, that it started with me ranting at the virtual bar at DevOps Enterprise Summit, but. We gathered a group of people together. Michael Nard is signed onto that. We had Rosalyn Radcliffe from IBM who’s contributed, who was a core contributor at the beginning of that. We had a lot of people from a lot of industries globally that’s translated into multiple languages. We’re looking for more translations

Murray: Yeah, it’s great. All right. Thanks very much for coming on Brian we, appreciate it,

Bryan: now. Thanks so much for having me. This is, this was a lot of fun. I appreciate it guys. 

Murray: That was the no nonsense agile podcast from Marie Robinson and Shane Gibson. If you’d like help with agile product development, contact At evolve.co that’s evolved with the Thanks for listening