Join Murray Robinson and Shane Gibson in a conversation with Jonathan Hall about implementing Continuous Delivery. CI/CD requires a level of test automation that seems impossible for most organisations. Jonathan describes a practical implementation path to CI/CD that everyone can do. Start by changing the role of QA to focus on pairing with developers to write good automated tests. Automate tests for all of your code changes, bug fixes and change regression testing. Automate your deployment and rollback tests. Test your changes one at a time against a fresh copy of production. Fix all defects and broken tests immediately. This will allow you to move to full CI/CD over time. And that will allow you to learn and produce results much faster and more efficiently.

Read along you will

Shane: Welcome to the No Nonsense Agile Podcast. I’m Shane Gibson. 

Murray: And I’m Murray Robinson. 

Jonathan: And I’m Jonathan Hall. 

Murray: Hi, Jonathan. Thanks for coming on. 

Jonathan: Thanks for having me. 

Murray: So we want to talk about an article you’ve written called CD Without ci. But before we do that, why don’t you tell us a little bit about your background and experience?

Jonathan: Absolutely. So I am a long time. Software developer. I was a full stack software developer. before that was a thing. I was installing servers running cables to them, writing the software that ran in those servers. Writing stored procedures in Postgres and backend Pearl Code and the front end JavaScript code, even before J Query was, known.

So I think, Full Stack pretty well describes that. And eventually as our company grew and we had people take over some of the operational stuff, we called them DevOps. Although looking back, it wasn’t really DevOps, it was just ops. But they provided VMS for us and stuff like that.

So I stopped running server cables and just focused on writing software. We hired a front end developer, so I became the defacto backend guy, and I’ve been comfortable in that space ever since, doing backend although with a heavy emphasis on infrastructure. I think what a lot of people would call a DevOps developer. If you wanna abuse terms a little bit. I do mostly go development in the last several years. And a lot of automation, C I C D . Setting up tools to make developers more efficient and effective. And leading teams mentoring to help people learn how to do these things that I’ve learned over the last 15, 20 years.

Murray: So what sort of companies do you work for? 

Jonathan: I definitely prefer smaller companies. In the last three or four years, I’ve worked heavily with fintechs startups. Not per se, because I have some big dream to work in a FinTech, but that’s who’s been hiring where I happen to live. I worked for booking.com for a while and there easily the largest company I’ve worked for. While I was there, I tried to introduce GitLab. We were using Git at Booking.com. Headless get so command line, everything, and I thought it’d be helpful if we could do some sort of code review. We were trying to do code reviews with certain third party tools. I thought GitLab would be great. It’s open source. It’s, there’s a free version available. So I implemented GitLab as a test project during a hackathon. I got minimal traction. I ended up leaving the company. I just felt like things were going too slow. Two years later I had former colleagues say, Jonathan, thanks so much for introducing GitLab. It’s changed our lives.

Murray: Ha Ha. 

Jonathan: Ha Ha. The, that plan to seed it and three years later reap benefits. That’s scale. I’m not at I to see results, in six months or less possible. So I, the smaller companies,

Murray: Okay, so continuous delivery. We’ve talked to Dave farley and Brian Finster about it, and both of them talked a lot about automation and said continuous delivery requires that you automate all of your tests and all of your build and deployments, but you are talking about CD without ci. So what is it? What are you talking about? 

Jonathan: Yeah, so I probably would retitle if I were to rewrite it again. But let’s talk about the problem I’m trying to solve with this approach, and then we can talk about the approach.

A lot of people in these companies will tell me that we can’t do CD here, we can’t do continuous delivery or continuous deployment because we don’t have automation in place yet. We’re still reliant on manual qa. We’re reliant on all these manual processes. Let’s work on automating these processes. And then once we’re ready, We’ll flip that switch and go to cd. And what I’ve just, noticed nobody’s ever ready. If you try to dedicate, say, 10% of your time to writing tests for existing code it’s a losing game.

You can never catch up. You’ll never have the test coverage that makes you comfortable or confident to switch over to continuous deployment. So I of flipped that around and I was like what if we start at the other end of the pipeline? What if we start at the delivery end rather than at the integration end?

And we start automating things there, and then we can use that as a framework to build the rest of our automation. And from there we can start to fill in the automated testing potentially replacing manual qa. I think that’s a great goal, but not everybody’s comfortable with that.

So I don’t always talk about that when I’m talking to clients cuz they’re afraid of replacing their manual So my objective with this approach is let’s get the end sight. Let’s get some light at the end of the tunnel so we can see the objective we’re trying to achieve, which is safe regular automated deployments, and then we can fill in the rest and use that as a guiding principle to fill in the missing pieces. That’s the level overview.

Murray: Yeah, I’ve certainly experienced the same thing that you have. I’m always talking about continuous delivery and test automation, build automation with people, and yet it’s frequently very difficult to get them to do it because they just don’t have a history of it. The developers and testers don’t know how to do it, and it seems like a lot of work that people don’t want to do. One time a few years ago with emphasis, I tried to get them to do it, and so they hired a separate team to write the automated tests after the development had been done, and then it was never updated, so it was completely useless. So I agree with you. It’s quite hard to get people to do it. So can you take us through that? How would you start, what would you do first and then how would you work your way up?

Jonathan: So the first thing I do is I try to take stock of the current situation. I advise people to make a list of everything that happens between the developer committing code, whether they’re using GitFlow and committing to a developer branch or something like that. Or maybe they’re using some sort of trunk based development or something where they’re committing directly to trunk, whatever the case from the time the developer hits commit until that code is in front of a live customer. What happens? What’s that process and how long does it take? The shortest I’ve seen typically is two to three weeks.

There’s usually manual QA happening there, somebody writing up release notes or documentation. Maybe a po signs off on the feature being complete whatever. My, goal is to formalize that first, and let’s let’s make the, make sure that stuff happens. First we all, agree what it is. And then let’s try to codify that before the merge button is pressed. So if you have manual QA happening, let’s say that takes three days, is there any way we can do that manual QA before we hit that merge button? And usually it’s not that hard. The hardest part is convincing people to think about the problem differently.

It’s not a technical problem. now there are some technical nuances that come up. if you have a large team and you only have one staging environment, how do you test all those features simultaneously on one staging environment? That’s a problem to be solved, but it’s a solved problem. It’s not rocket science. It’s just a matter mainly of communication. So the goal is to find all those processes, document them, put them in a template and make it a check mark for each item. So the first step isn’t to automate anything at all. It’s to document what you’re already doing. 

And then. As you can. If any of those things can be automated, do that. Stop waiting on manual processes, but more important for this process, just move that into your definition of done, and do as much of that as possible before that merge button is pressed. That’s the first step. 

Murray: Yep. Is this the typical deployment pipeline that you’ve talking about in your article? So pull request, merge to dev, goes to qa, merge to staging, acceptance testing, merge to mainline, docs deploy. So we’re talking about making sure that’s working properly first.

Jonathan: Right. The illustration you see in the article, was based on my experience at an e-commerce company I was working at several years ago. But I think it’s typical especially using GI flow. Not every company has exactly the same list of steps or in the same order, it’s fairly typical.

Shane: So what percentage of organizations or teams , you work with? Have manual QA teams.

Jonathan: I would say close to a hundred percent either have it or think they should have it. Sometimes I start early enough and it. they’re like, we’re still getting ready to hire our first qa, and I think that’s gonna get them into the big league somehow.

Shane: So that intrigues me because, there’s been an mantra of bring testing into the team And then mentor and coach the rest of the team on how to do those things that you do. Figure out how to automate things get the developers to build the happy path tests. And then you start looking at the edge cases. And we’ve been talking about that for years, but it sounds like there’s a really small amount that have got to that nirvana of bringing automated tests in and removing that manual task. 

Jonathan: I agree. The common attitude I see is that automated testing is great, but we’re not there yet, or we’re not big enough for that yet. So we’re gonna keep doing manual testing. There’s this idea that yeah, if you’re Google, then yes, automation’s great, but we’re not Google, we’re still only have six developers or whatever. That’s part of my mission is to change that mentality. You can have big tech success on a small tech budget. You don’t need to wait to start doing the right practices. You can do them now.

Murray: Yeah, my experience consulting in the Agile space, Shane, over the last 15 years, is that every company I go to has manual testing and it’s often separated from development. So the first thing I’m doing is bringing to gathering the same team. There have been a couple of companies in the last five years that do have quite a lot of test automation, but the test automation has always happened at the end. It’s not happening during development. It’s happening as part of developing release packages and things like that. And then sometimes it’s maintained and sometimes it’s not. So it gets put in places like a final gate. 

Jonathan: I definitely see the same pattern. 

Murray: Yeah. So it’s pretty terrible. And I’m still hearing this thing of you can’t have developers test their own code because you need an independent group to look at it. Cuz developers won’t take care or they’ll just test their own assumptions. 

Jonathan: Nonsense. 

Murray: I agree. It is very much about justifying the test empire for the test manager. And only Mike states to have a separate test team. When you have a poor quality development team, as you sometimes get with outsourced off shore teams, but even then the real answer. Is just to get some better developers. 

Jonathan: Yeah. There’s a place for testing and there’s a place for manual testing. It’s just that it shouldn’t be in that delivery pipeline. That’s the wrong place for it. Who wants to hire a plumber who doesn’t test the toilet he just installed before he leaves? That’s just ridiculous. And in no other industry would you trust a professional to do their job and not test it before they send you a bill? That’s just nonsense. Why? Why would developers be different? We’re supposed to be the smartest people on the planet. Why are we so dumb?

Murray: That’s a little unfair, Jonathan because developers do test their own code, but they just don’t test it thoroughly enough. Normally. And I think Continuous delivery is saying if we get people who are really good at testing and people who are really good at development together and we get them to overlap, then you know, you can get some magic that happens. You can automate your tests as you go, but it’s still uncommon in my experience. 

Shane: It depends on the culture of the organization. So if the developers are empowered to get the job done and iterate their ways of working, then good developers will start to automate manual tests. They’ll write code that tests their code, but if they’re in a feature factory where every minute they work is watched by somebody else and they’re micromanaged, they’re never enabled or allowed to actually go and automate their work, they just get told to do that next feature and pump it out. So often the culture of the organization is a constraint to some developers For doing what they know is something that’s valuable. 

Murray: I wanna modify what I said before because I realize that there’s a lot of unit testing being done by developers. About 10 years ago, a lot of unit testing came in, and that gets run before a build, but that doesn’t actually catch that many defects, I, find. Yeah, it catches the micro defects, but it’s really the integration test, whether it’s manual or automated, that catches the big problems.

Jonathan: So I have a few things to say on those topics. The first is I often see teams where they’re doing practically zero unit testing also. That’s a problem to solve that doesn’t require manual or automated key way at all. It just requires good developer practice. Second I do believe that you can catch a lot of bugs with unit tests if you do the unit testing well. There’s a lot of bad unit tests out there that only test a happy path, for example. Or just don’t think about a lot of the things. An interesting finding came out the book Accelerate that few years ago. One key finding was that teams that don’t have developers writing their own tests do not see better business outcomes. If I some code or a feature, and then someone else, whether it’s QA or a different development team or whatever, writes the tests, those tests may as well not exist from the standpoint of business outcomes. 

I use an analogy to explain why I think this makes sense. And that is what I cook in my kitchen for my wife and I leave a disastrous mess around. My wife has two choices. She can either tell me to clean up the mess or she can come back after dinner and clean up the mess herself. When I’m forced to clean up the mess myself, I’m less likely to make that sort of mess again in the future because I’m thinking about it. I don’t like having to clean up my mess. So I’m gonna try to make a smaller mess the next time. Even subconsciously, I’m probably thinking about that process the next time I’m preparing a meal and I’m less likely to use six spoons instead of one or whatever. I think if we apply the same principle and have developers write their own tests and of course also fix their own bugs they’re less likely to make those same mistakes in the future. It makes sense to me intuitively. I don’t know if that holds up to the science, cause I don’t think that science has been done yet with regard to testing. We do know, or at least we have strong evidence that if developers aren’t writing their own tests, you might as well have those QA people playing doom all day for the amount of business value it provides. 

Murray: Okay so imagine you’re going into a company. They haven’t automated much. You talked about defining your process, so how do you increase the quality and, get them on the path? How would you see Devon test working together?

Jonathan: I think there’s two parts to this answer. The first is, what is the ideal scenario look like with devon Test? and in, in my view, the ideal scenario, where we’re trying to go doesn’t necessarily have Devon test quote on the same team. It might, although that’s not the important feature. The important feature is what is test, or what are qA doing, and what is Dev doing? And so I think the simplest way to describe my ideal scenario is that QA testers, manual QA testers should not be doing regression testing. They should be doing exploratory testing. You have QAs exploring the system to find what’s unintuitive or what’s broken. This is valuable and this is stuff that automated tests cannot do. On the other hand, if you have your, manual qa, people just reading a script of these are the 16 steps run these 16 steps, why wouldn’t you automate that? That’s the sort of thing computers are good at. They do it better than humans do because they don’t make mistakes. They don’t forget step 12. So if you have your manual testers doing manual regression testing, stop, you’re doing your product a disservice. You’re not getting the quality you need because human testers are unreliable.

You’re doing your testers a disservice cuz they’re bored. You’re doing a developer a disservice cuz they should be automating those tests. Everybody involved is not getting the best benefit. If you have your QA running scripts, computers do that stuff. Now there’s a huge debate that could be had about whether that should be done in the front end or the back end unit test, integration test, whatever.

I think there’s a different conversation, but the point is, it should be automated. Have your manual QA people doing what they do well and exploring using their brain and their intuition to find problems that you cannot find with regression tests. In this way those manual QA people are closer to user research or UX design which are incredibly valuable and just simply cannot be automated.

Murray: Yeah. And also if they’re doing that sort of exploratory testing and they find a problem, then you can develop a regression test for it. 

Jonathan: Absolutely. And you should. I’m an advocate of tdd. The first thing you do when you discover a problem, write that regression test then fix 

Murray: Yeah. I agree. All right The first thing we are gonna do then is get people to automate their regression tests. 

Jonathan: Yes. 

Murray: And how would you go about doing that? 

Jonathan: Great question. The first thing I would not do is tell the developers, go back and write tests for all the code you’ve written. That is a huge waste of time is demoralizing, and it’s even dangerous. Often. What I do what I advocate is a working agreement on the team that any new code changes get tests. Whether you write the test first or after you decide. I’m an advocate for tdd, but a lot of people don’t like it. And I understand why. I’m not gonna enforce that, at the beginning. But if you’re as a developer, you’re fixed to bug. Make sure there’s a test that proves that bug has been fixed. Whether you write it first or second doesn’t matter. If you add a new feature a new dropdown to a menu at a test to make sure that dropdown works as expected if you apply this rule of all new changes, get tests at the time they’re written, and you don’t approve that poll request, for example, until the tests are there. Then eventually you’re gonna have all of the important code tested. By important, I don’t mean to suggest that the code that’s sitting there and not being changed isn’t important, but it’s not a liability. It’s not important in the sense that we know it’s working, it’s being tested by our users in production already, presumably. So you don’t need to go back and refactor it, which is dangerous to add tests. Just leave it there. Add tests when you change it because you found a bug report in the wild, or because you had a new feature?

Murray: But what about the thousand manual regression tests that the test team does every few months when you’re gonna do a deployment, should you hire some sort of low cost offshore team to automate all those for you based on the old manual scripts?

Jonathan: Probably not. What I have done with success in the past is I keep that process in place at first. So again I was describing in the beginning, find your manual processes, move them to before that merge happens. And if that manual process includes regression testing by humans don’t change that yet. That’s not the first thing to change. So keep doing that manual regression testing, but I would suggest not doing it every three months. Do it every time you merge a feature. 

Murray: But we can’t do that cuz it’s gonna take two months to do it. 

Jonathan: I get that exact same pushback all the time. But if you can find a way to do your manual regression testing per feature, a few magical things start to happen. First, you get feedback on that feature faster, rather than waiting three months to discover that feature is broken. Maybe it still takes three days, but that’s better than three months, so you can get your feedback quicker. And maybe the most profound thing that you discover here is you start to see where your actual bottlenecks are in your process. If it takes three months to get through featured by feature regression testing, then that’s a, big sign that you have a problem to solve. And that problem is often glossed over if you batch everything together and test it all at once. And a third one, maybe a little bit more obvious one is when you do feature by feature regression testing you’re testing each feature in isolation and you don’t ever have to wonder, wait, was this Bob’s feature or Alice’s feature that broke this thing? The only change on this branch that I’m testing was Bob’s feature. So that means that he broke it.

Murray: Yeah, I think it. makes a lot of sense to automate your aggression test on Deltas on feature by feature. I’ve done that as well. I’ve encouraged the team to build that into their process. So let’s say you’re doing that. What does your process flow look like and what environments are you using to do this? Cause you’re not going straight to production here.

Jonathan: Generally, no. So every team I’ve joined have a production environment. They usually have at least one, sometimes two tests or staging environments. And this is usually where QA happens, so code gets merged to this, let’s call it staging environment. Manual qA happens there for the course of a few weeks, and then it gets promoted to development or to production. So what I usually propose as a first step is to break the chain between development stage and production. So that stage environment is a dead end. So you can push to stage anytime you want but every time you do, or almost every time, it’s gonna be a force push, it’s gonna override whatever’s there. So you’ve created a feature branch. You’re working on feature X, you’re ready for QA to test it. You force push that to the stage environment. So now stage represents only your change. It’s whatever’s in production plus your change and nothing else. There’s no other 16 features there that might conflict. QA does their testing and they report back to you, and I tell you that everything’s good. You’re good to go. Then you merge it into production or they tell you there’s a problem. Then you go back to your feature branch, you’re working on your thing. Meanwhile, Alice working on her feature, why force pushes to stage QA does the testing on her feature and so on. So there’s clearly a bottleneck there because stage can only be used by one feature at a time. But this goes back to what I was saying earlier about discovering your bottlenecks.

Shane: So effectively you are introducing a forcing function, which is push it to stage. Nobody can move until QA finished their job. And then the next one’s up. So we’re looking to focus on how long is it taking to get that work done? And if it becomes a bottleneck, what can we do to reduce the time cycle. By making it a constraint then all the workarounds we normally do to get It through when we haven’t tested it properly or when we’ve got a feature clash we can’t do it anymore, because we’ve got single line of sight and it stands out like dog bollocks when the process isn’t working. 

Jonathan: Exactly it’s a technical whip limit Of one. Now maybe your team’s big enough that you need two, so you can set up a second staging environment. That’s a simple next step in for many teams. Other teams will set ephemeral environment such that every pull request creates a new temporary environment. If you’re using kubernetes or something that’s fairly simple to do, or ECS or whatever, you can do that too. But having that forced whip limit is really nice. A lot of times, especially for a small team, it and it really shows you where your bottlenecks are. The alternative, which everybody does when you have everything batched is, oh, it’s time to release.

We think is done testing. You haven’t really thoroughly tested everything so you know, which would you rather have slow down your work or only half test stuff. That’s the option you’re choosing between.

Shane: And I can see though that if we carry on working the way we used to work, but introduce this idea of single line of site staging, but we do introduce two staging environments. We have to be really careful that we are not forcing our QA team to time slice across the two staging environments because we’re not actually fixing the problem then. We now have a work in limit progress of a whip constraint of two, but a team of one, and so we haven’t actually fixed the problem, yep.

Murray: Let me describe the approach. I see agile teams use a lot, and then you tell me what’s wrong with it. They’re working on a project. They’re doing a bunch of things together that they expect to release together. Somebody has a small feature to do. They write the code and a unit test at the same time. It goes through a code review with the other devs in their team. Once that’s been approved, it gets merged into a team test environment where the testers in the team test it straight away and find problems and then send it back to dev to fix which maybe a couple days delay. And then as it looks like it’s gonna be okay. The testers might work with the dev to update some integration tests because if they’re finding bugs and they weren’t being picked up in the unit tests, they’re probably integration somewhere. So they build their integration test suite together at the same time, and that’s sitting more in the team test environment.

And then once that’s working, it goes to acceptance testing with the product owner. Who just does their manual stuff and some SMEs do some stuff and they probably find some more issues and, so it goes back to dev and test, and they update their tests and then it sits there in this acceptance test environment waiting for a few more features to come along. So they’ve got a package to release as a product owner doesn’t wanna release the features one by one. They wanna release them in a package so that they can communicate it. And then as they get ready to do their package they run all of their automated regression tests again. They do a bit of exploratory testing, and then they’ll, which hopefully only takes a couple of days and then they release their package. So I see a lot of that from teams that think they’re pretty good. What’s wrong with that? 

Jonathan: Several things. What you’ve described though, sounds very close to the team I was working on when I wrote this article. We had devs and testers on the same team. The problem I observed on this team I was working on although the devs and QAs were on the same team, there was still a delay. And very frequently the first half of the two week sprint was devs hacking away while the QA was idle, waiting for something to do. The second half of the sprint was QA testing like mad while the devs were sitting there waiting for something to do. So that’s obviously not very efficient. We haven’t actually torn down that silo. We’ve just localized the silos if you will. So now we have silos within a team. That’s not really much better than having a separate QA team.

Maybe it’s a little bit better cuz maybe the cycle time is days instead of weeks, but it still hasn’t fundamentally solved the problem. So what I aim for here is to put QA in a advisory role. I really like the metaphor from the Toyota production system of the use shaped cell which is rather than having a long assembly line that’s just a conveyor belt that starts at A and ends at b.

You have a U-shape conveyor belt that starts at a and ends at a with a workman standing in the middle. And they have all the tools they need in this U-shaped system to get their piece completed. Whatever they’re doing. They’re welding pieces together or they’re building a widget, whatever. They have all the tools they need arranged in this so that they have easy access. They stand to one place and pivot around and do the stuff. They pick up the part from a bin, they do the stuff and they drop it off in a bin. I like to use this analogy for development and QA in this analogy becomes one of the tools on this U-shape bench. So rather than QA being a process the traditional view is developer takes a feature, a story, they do some work and then they hand it off to qa. QA does some work. They either approve it to go on or they pass it back to the dev. Let’s stop doing that and instead make QA an enabling force a tool that the developers can use.

Just like many other tools, if you need a database or you need an IDE or whatever, these are the tools the developer uses to get their job done. So the developer works on his feature when needs assistance to validate that it’s working correctly. He gets help from qa, and that help can come in different forms.

It can be that the QA team or person has built a platform that automates Selenium or whatever, and they just teach the dev how to use that. It could be the QA sitting next to them and doing a pairing session where the QA is showing, here’s the best way to test this thing. It’s to new feature. You don’t know how to do this, here’s the way you do that, et cetera.

But QA then becomes an enabler rather than a step along the delivery pipeline. And this eliminates that waiting time where QA is waiting on dev, and Dev is waiting on QA because it’s now a pulling function. The developer says, Hey, qa, I need you. And they pull them in to help for something briefly, assuming you have the capacity on your QA team to do that 

Murray: So QA in this scenario really becomes quality assurance in that they focus on the process rather than actually executing the tests at the end 

Jonathan: And this gets back to the, thing from the Accelerate book that the developers should be writing their own tests here. The key way just helps with that. If the key way has expertise, the dev doesn’t have, they share that expertise, but they don’t do the work for the developer.

Murray: yeah. So they say things like, how are you gonna test this? What’s your rainy day path? What’s your integration test? Have you thought about this? Have you thought about that? Did you know that there’s these other integration tests over here you could be using? What data are you using? Yeah, dev test pairing. I think very helpful. 

Jonathan: Oh yeah. 

Murray: I tried to implement this once where part of the team was offshore in a developing country for cost reasons. And we had terrible trouble. The offshore team complained a lot and said, just give us detailed test scripts and we’ll execute them. That’s what we like to do. That’s what we wanna do. Just do that. What’s your experience with doing this with those sort of offshore teams. 

Jonathan: Yeah. My answer’s not gonna make very many people very happy, but don’t do that you’re not doing anybody favors. You’re not saving the cost. You think you’re saving. That’s pretty close to the scenario. Actually, we had at the company that inspired this post. We had two offshore teams, in two different countries. And our QA was in one of those countries. And ultimately we hired an internal. Freelance qa automation engineer to help set up a testing platform. Which is a six month project. And then we essentially fired our external QAs, and we prepared for the worst. Everybody was nervous. We’ve been doing manual for years. What’s gonna happen? The customers gonna start complaining, quality’s gonna go to crap until we figure out this automation thing. We had one permanent internal QA who was gonna stick around to do what I just described. Be the helping hand.

And then the, our freelance QA engineer who was there to provide guidance at the beginning, we set up recurring meetings, to address all the fallout we knew was gonna come when the manual qa team of four was let go. Nothing happened. We had zero incidents, we had zero quality reports. Nothing happened. We were bored. I’ve had the same thing happen again at another company more recently. This nuclear disaster we were expecting it doesn’t actually happen. So there’s two possibilities I can imagine. One is maybe some quality was lost and it just was lying in weight and it would bite us six months down the road.

Or the quality assurance that our QA were doing wasn’t actually that valuable. And there’s probably a combination of both of those, but it’s not this disaster that I expected or that we expected. Maybe that’s different if you’re in a life and death situation, if you’re building self-driving Teslas or something.

Maybe that’s a different scenario. But it hasn’t ever bitten me to switch to this new approach and stop depending on manual script executors. So just stop doing that. It’s not building a better product. There are far cheaper, more effective ways to build quality of your product than to expect somebody to inspect quality in later

Murray: Yeah, I agree with you. That was my experience and my recommendation will you get rid of the offshore test team? But they’re so cheap. They’re Jonathan. They are only $200 a day when people locally are a thousand dollars a day. Surely we are saving tons of money.

Jonathan: In a absolute sense you’re saving money versus hiring that a thousand dollars a day person, but what if you could save all of that money and actually build quality into your product instead? 

Murray: Yeah. But then I won’t have as many people on my team reporting to me, and Yeah. To justify my salary. 

Jonathan: If your goal is to have more direct reports because it makes you feel powerful, then by all means hire manual keyway testers. But if your goal, on the other hand is to produce valuable software then I can help you

Murray: Yeah. Okay. So in the process, you are talking about what’s the kind of cycle time to get things into production. 

Jonathan: So my goal is to get things to production as quickly as possible. That’s of the goal of continuous delivery or deployment in the first place, I have easily gone from three to six weeks cycle times down to 150 deployments a day in the course of three months. 

Murray: How many deployments a day? 

Jonathan: 150. Now we’re talking About a microservices architecture. So I don’t mean the entire platform is deployed in a day, but some services upgraded. There are 150 service upgrades in a day on a team of 10 people. It’s not hard. Not everybody needs 150 deployments a day. But the point is, once you have your continuous delivery in place, why wouldn’t you deploy?

We could talk about that if we want. There are reasons not to make new features accessible to customers that often. You wanna do some sort of PR announcement or Write the documentation or, you wanna batch things up. If you’re developing mobile apps, you certainly don’t wanna send 150 updates per day to the play store. There are technical and business reasons not to expose that many changes to your customers in a day. But there’s no reason you can’t have your continuous delivery pipeline build that artifact that frequently.

Murray: I just wanna go back to what you said about only one change on staging at a time. If you got one change on staging and it takes a couple of days to test it, how on earth are you doing 150 changes a day.

Jonathan: First you’re not making changes that big. You’re doing changes that don’t take two days to test. By the time you’re doing 150 a day, most of your changes don’t need manual QA at all. Maybe it’s a simple typo fix. Maybe you’ve, you had an IF conditioner that was wrong, you reversed it. You don’t need a manual QA necessarily to test that. You can test that with the unit test and you’re confident. Many of those changes are gonna be small changes that don’t require manual QA. But the big ones know, you’ve added a new feature of some sort that does require some sort of, manual qa cuz you don’t have the automated integration test yet.

Yeah. Then those cases do go to maybe two day long testing process. the point is that you shouldn’t have everything in your pipeline waiting on those big features. You can let your quick things go through quickly and your slow things can go through slowly.

Murray: Yeah, and every time you start by banking or staging a copy of production, and then you’ll apply your change to it and check it. Before you deploy it to production. Yeah. Okay, good. So this is really an implementation path towards cd. A step by step way to getting to CD when CD is too intimidating and too scary people. 

Jonathan: I would never advocate to somebody implement the automated deployments and stop if you do that you miss the whole point. That’s a starting point. That’s to get the end insight and to help make clear where you need to improve which parts are the most painful and should be automated first And things like this.

Murray: Okay, so if I’m understanding you correctly, the implementation path goes something like this. First automate your build and deployment process with your rollbacks and your database backups and recovery and that sort of stuff. 

Jonathan: Correct.

Murray: Then start automating your tests on your delta changes that you are doing. And do those by bringing testing to the front. Turning your testers into QA people. And that might require different QA people. We might be talking about QA engineers now, people with a much higher level of skill, so you might need to get them in. And as doing that, then you are transferring the skill to the developers so they can do it themselves really just for somebody helping them and advising them. And then as you are making all these delta changes, you are automating more and more of your regression tests until over a period of time. Maybe it’ll take you a couple of years, but you’ll have everything automated and then you’ll be CD at the end. 

Jonathan: Exactly. Yeah. It sounds so easy, doesn’t it?

Murray: It does sound a lot easier as an implementation path than going straight to cd. Every change has to be deployed directly into trunk. It’s very scary for people who are not there. 

Jonathan: It definitely is. It’s uncomfortable. And it’s scary and aside from the technical changes, it requires retraining your entire team of all at once. Alright, everybody, we’re gonna go through this bootcamp about the new way of working. Change everything about everything you’ve done, and now we’re gonna be better. That’s a really hard sell. Emotionally, technically, everything. It’s just very difficult to do that. It’s easier if you can pick off the pieces that matter most to your team and your context and do those piece by piece. 

Murray: Now you’ve been having a few conversations with Brian Finster about this. What does he think about this approach? 

Jonathan: Yeah, we talked about this a bit. He convinced me that the title of my article is Wrong. I call it CD without ci. And he has co-authored minimum cd.org, which is a great resource if you’re looking into cd. It outlines the minimum requirements to say that you’re doing cd. And according to this website, which I actually agree with if you’re not doing continuous integration you’re not doing cd.

If you’re not continuously integrating your code to Maine, how can you continuously deploy it? So I’ve renamed this approach to just call it Lean CD. That’s the name I’m using these days to describe a lean approach to adopting cd.

And I try to be very clear when I’m talking to people about this, that this is not the end goal. If you do this, you can’t claim, now we’re doing cd, we’re done. We can go home now. This is the starting point. All I’m trying to do is change the way we think about the problem so that we can use a different starting point rather than trying to, swallow an elephant all at once of automated everything and now we’re done. What if we could change the goal post a little bit, change the order of things so that we have a clear view of what we need to do on our team and so we can start seeing results immediately rather than maybe in three years?

Murray: Yeah. It feels much more practical to me. What do you think, 

Shane? 

Shane: Yeah. I like the idea of starting small. Focus on one problem at a time. Solve them, solve the next one. Get good at that. The whole boil the ocean, the whole stop what you’re doing we’re gonna refactor everything for six months before we start adding value back to our customer. Yeah. That’s hard to justify. Gonna stop the whole team and they’re not gonna deliver anything out to production for six months because we’re gonna build a new process. Good luck with that one. 

Jonathan: And it’s pretty ironic that we would have this conversation in the quote, agile circles, where we’re all about doing incremental changes. Why would we suddenly do this in a big bang approach?

Shane: We still have the term agile transformation, where the goal is to be agile and not to actually use agile patterns to change the way we work to add value, so it’s that whole idea that it’s a project, it’s a thing you do once and then you stop doing it. So yeah. I like the idea it’s incremental, I like the idea that we focus on the next problem to solve. We get feedback that, Hey, that didn’t work. Let’s not do that one again. Yeah. It’s a good way of rolling it out. .

Murray: What sort of outcomes have you seen as a result of doing this? If I was the business person, the product owner, or a project manager or something like that, how would you persuade me that this was a good return on investment? 

Jonathan: I’ve had to do that a number of times naturally. I was actually recently helping a team that was struggling with slow releases and the product manager wanted to introduce a new stage to our deployment process make things slower because this would allow a new cue to be formed. I was like, what if we did a different approach? And she was like, I don’t know, but let’s try it. We did try it. What was the result? Faster feedback. We weren’t waiting weeks. So here’s A really good real world story of a tangible result. This team that I was just describing was having some database performance issues and we needed to fix this. And the current plan was let’s replace the database back in with a new database that’s faster. And that might be a good long-term approach, but it’s certainly not an incremental short-term approach. It was gonna be a month long project to do this. So I wanted to put some profiling into the code base.

You, I was like, where can we find the slow queries? What’s the slow parts? What’s taking so long? If we had gone through the normal process, it would’ve taken three weeks to get the metrics from the system, and then three more weeks to make a change that might help, and three more weeks to make the next change, and so on and so forth.

That was one of the catalysts. That was one of the arguments I made on this team. If we adopt this new approach, we can still spend three weeks testing features if we want to, but at least these features, these changes can be fast tracked without having to resort to the hot fix approach or whatever. A special approach. So we did that and then within three days we had the database problem solved. No, not permanently. They still need to refactor the database and replace the new one, but the immediate problem, the problem that was causing customers pain was solved in three days because we were able to get those new changes into production quickly and verify them and learn from them and solve customer problems. That’s the reason that continuous delivery is so powerful in the first place. It’s fun to deploy things fast, but the reason the business cares, the reason we care as an industry is so we can serve customers needs faster. And you can’t. So that the bottom line.

Murray: Yeah, and one of. Big benefits of continuous delivery is that we can learn very quickly from our customers what works and what doesn’t.

Jonathan: exactly. 

Murray: Yeah. So it’s optimizing for learning. All right. Should we go to summary Shane, 

Shane: Yeah. I like this idea that you started off with that nobody’s ever ready. We should never wait until it’s a hundred percent done , just ship it early and get some feedback and change what doesn’t work. So I like that approach and pretty much everything. The other one I’d like is this idea of mapping out all the steps first, observe what you’re doing, document what you’re doing. Treat it as nodes and links so you can see the supply chain of how you work.

And once you’ve done that, you’ve got a baseline. You’ve got all the steps that you need and sure as shit, you’re gonna find some surprises, there’re gonna be some tasks in there that people dunno are being done and will start questioning why do we do them? Typically the answer is cuz that’s the way we’ve always done them.

But there’ll be some surprises in there. There’ll be some early ones. Then flag every step that’s manual. So once you’ve done that, then just start figuring out which manual process you can automate next. Don’t do them all at once, just pick them off one by one. So I like that. I like your comment that unit testing is just good development practice like you said, I expect my plumber to flush the toilet after he is fixed it.

I don’t expect to package it up and send it offshore or have a virtual team tell me to flush it for them while they watch on a camera. There are certain things that are part of our definition of done, they are just. Reasonable practice for us as professionals. And we need to remember that.

I love the fact you use food references. I’m a great fan for the whole cooking thing. I like that idea of keep your own kitchen clean. If it’s too messy, you gotta do the scrubbing at the end of it. Yeah, clean up your own mess. I like the other idea that QA skills are still needed for the tests that require human exploration, there are still things that we can’t automate and we still need humans to do that. So let’s let them do that. They shouldn’t be focused on the Excel run sheet of click here. tick here. We can automate those and we should over time. I like the other point you made about when we’re starting out, don’t force you to have team to go back and retrofit test to every piece of code that they or somebody else wrote over the last 10 years.

Never gonna happen. Never gonna be finished. Low value, piss them off, make an unhappy team. They’re not gonna start adopting automated tests if you make them go do that rossy work at the beginning. You fix a bug, you write a test. Yeah. And you’ll get coverage of the important things over time.

And I like that idea of feature by feature testing. So break it down to small chunks and the forcing function. You talked about that. It gets put up, it ever gets passed or declined. And now we’ve got a bottleneck. Now we’ve got focus on getting that process automated or faster because we’ve got a blocker and it’s a visible blocker, so everybody’s gotta work to unblock it.

And I’m a great fan of the idea of QA as coaches bringing those constrained set of skills into coach the rest of the team, how to do what they do and then look for those edge cases and automation. So yeah, they’re enabler, not a step in the process as the word you used. And so I think that’s a good way of describing what that role becomes in a self organizing, self sufficient team. So that was me. Murray, what do you got? 

Murray: I think continuous delivery is really scary for most organizations. it assumes a really high level of technical skill, and capability and that you’ve fully automated everything and you’ve fully automated all the time. And even organizations that would say that they’re agile are nearly always a mi million miles away from that. It’s probably only the really good organizations that are there. So how do you go from where you are to that place? And I think what you’ve done is you’ve laid out a practical path that people can take. We’re not gonna say that you’re gonna do manual continuous delivery.

What we are doing is, we’ve got an implementation path where you go step by. automating your build deployment rollback process using all the usual tools. And then you change the role of testing in your team to be a real quality assurance role. And the goal is to move away from separate manual testing except for exploratory testing to developers develop automated tests as they could, but ones that the QA people using their experience would say, yeah, this makes a lot of sense.

You’re testing the happy day path, the rainy day path, the integration problems and UX problems. So the role of QA then is to pair with developers at the beginning and during the process to agree on what sort of tests are needed to be automated. And then just to really just to make, probably just double check that the developers are doing that because you don’t want testers to do manual regression testing.

So this is our goal here. Get rid of manual regression testing. The only testing that these QA people should be doing is exploratory testing to find unexpected problems or where there’s something that’s got through that you didn’t expect. And then once you’ve found them, then they can pair with the developers to automate those tests.

So I really like the idea of developers and tests is pairing together to automate tests. I think that’s a really good solution because then they can learn a lot from each other. Now this may require you to have quite different sort of testers in your team. Testers who are much more highly skilled, that are really good at automation testing and that their role in the team is to help the team develop their own automated testing framework and to do this moving testing to the front. I think if you do that, you’ll save an enormous amount of time in your development process. Your cycle time will probably be cut in half , or lot more. I think you’re gonna dramatically improve your effectiveness and efficiency of your team, cuz they’re not gonna waste a lot of time trying to find bugs because you’ve automated a lot of them out. So the bugs just don’t come up a second or a third time after you found ’em the first time. And if you keep doing this, you’re gonna get to the point. Everything is automated and you are at CD. You don’t have to get there straight away. We’re just gonna go through this process, automate our deltas focus on changing the role of testers to real quality assurance, not testing really.

And I think that’s great. I like the idea of just having one staging environment and having it always be a copy of Dev plus the Delta of whatever small thing you’re gonna do . That I think place is quite a strong limit around how far you go before you deploy something.

I like it. It feels to me like a practical path to cd. You’re not at CD yet while you’re doing that, but you can get to CD by the end, and you can get a lot of benefits as you go along the path. That’s my feeling about it. Did you want to comment on anything we’ve said? 

Jonathan: I think you guys have both summarized pretty well. Of course there’s a lot of room for nuance and questions which I’m happy to discuss if anybody’s interested. I would like to. leave maybe with one concluding thought and that is that 10, 15 years ago, a discussion like this would never have happened. We would not be discussing how to implement cd. We would be discussing what is cd? Is it even theoretically possible to deploy 10 times a day? Now we all know that it’s possible now the question has moved to can and should we do it here? And I think we’ve made huge progress as an industry. Part of that’s technological, computers are faster, storage is cheaper et cetera. But it is mostly really a mindset shift. And I’m really glad to live in a time when we can have this level of conversation about what’s the best way to implement cd rather than me trying to convince somebody that it actually is possible to deploy 10 times a day if you want to. That’s a boring conversation now.

So I’m really pleased that the conversation has moved as an industry and I think we’re making great progress and things are only gonna get easier and faster. 

Murray: All right, Jonathan, where can people find this article we’ve been talking about? 

Jonathan: Yeah the articles on my website, j hall.io. That’s j h a l l.io. You can find the article there. Search for CD without ci. I blog daily, so you can read a lot of stuff I write there. I have a podcast on the website too about DevOps for small companies. And feel free to reach out. 

Murray: What’s the name of your podcast?

Jonathan: Tiny DevOps, focus on DevOps software delivery practices for small companies.

Murray: Excellent. All right. Thanks very much for coming on, jonathan. 

Jonathan: Thanks for having me. It’s been a pleasure. 

Murray: That was a no nonsense. Agile podcast from Murray Robinson and Shane Gibson. If you’d like help to build great teams that create high value digital products and services, contact Murray evolve.co. Let’s evolve with a zero. Thanks for listening.