From the Wild West to Agile and Beyond with Jim Highsmith

Join Murray Robinson and Shane Gibson as they chat with Jim Highsmith, one of the authors of the Agile Manifesto. Take a journey through the history of software development, from the wild west to structured methods and the emergence of the agile movement.

Key insights in this conversation include:
🔸 The agile mindset and the problems it solves
🔸 Insights into what’s missing from the Agile Manifesto
🔸 The problems with the ‘Agile Industrial Complex’ and the SAFE framework
🔸 The issue with life coaches taking over agile coaching
🔸 The future of agile leadership

Tune in for a fascinating discussion with one of the founders of the agile movement and gain valuable insights into the evolution and future of agile.

Read along you will

Murray: In this podcast, we talk to Jim Highsmith, one of the authors of the agile manifesto. We discuss the history of software development from the wild west to structured methods. The agile movement and the courageous leaders who implemented it . We talk about what agile is. The agile mindset. The problems agile solves and why it took off. We talk about what’s missing from the agile manifesto and what was harder for the industry to understand than people thought. We discuss the problems with the agile industrial complex. The problem with safe. And the life coaches taking over agile coaching. And finally we discuss agile leadership and what comes next? Join us for a fascinating discussion with one of the founders of the agile movement. 

Shane: Welcome to the No Nonsense Agile Podcast. I’m Shane Gibson. 

Murray: And I’m Murray Robinson.

Jim: And I’m Jim Highsmith. 

Murray: Hi, Jim, Thanks for coming on. 

Jim: Thank you for having me.

Murray: So Jim is one of the authors of the Agile Manifesto and has written a new book from Wild West to Agile. So that’s what we wanna talk about today. Jim, why don’t you kick us off with a summary of your career.

Jim: I did electrical engineering in 1966, and my first project out of school was to work on the Apollo Moon Space program. I got to work down at Cape Canaveral on some ships that were gonna go out in the ocean and track the Apollo spacecraft. In that time, computer science was way in the distance, and it was just a basic electrical engineering. We still made circuits with transistors and resistors as opposed to integrated circuits. 

I decided that I wanted to be more general, so I got a master’s degree in business in 1970 and went to work as a systems analyst. At that point in time, You were an analyst, you were a designer, you were a programmer, you were a tester. And I even was a computer center operator for a while.

So in the good old days, the interaction with the computer was you punched up your cards, you put ’em in a card reader, you ran your computer program, and about 12 hours later you got printout.

So I then had a job as a manager of software development for a utility company in Atlanta, Georgia. And I got to build my own staff during that period and decide how we were gonna go about software development. And this was the mid to end 1970s. And so I started looking at structured methods and brought those into the organization. then I had an opportunity to go with a small startup. So I went from a big safe company to an unsafe startup in Topeka, Kansas.

All during the eighties, I did structured methods, I taught ’em, I sold them, I installed them, I did, transformations for people. And then towards the latter part of that decade, we started building what I called monumental waterfall monumental methodologies. And those persisted in through the nineties. In about 1990, after having done structured methods for 10 years, I began to think, yeah, but this isn’t working very well And this is not actually the way I personally work. And IT managers and executives were saying, I T costs too much, takes too long. It doesn’t give me what I want. And so the nineties were a time of what I’d call rapid application development. In 1992 or three, started working with a computer manufacturer doing a rapid development tool and the marketing manager and I came up with an iterative methodology. To show how this tool worked and we would build applications in a month using one week iterations. As the decade wore on, I got more and more into rapid application development.

And then in the latter part of the decade, I was writing a book called Radical Software Development. It’s interesting how I got involved with the Agile movement. It was a combination of Martin Fowler and Kent Beck. But we ended up in Snowbird writing ad Agile manifesto. And then from that point, I really spent the next 10 years working with companies on Agile projects both big and small both software and hardware.

And then 2010, I went to work for ThoughtWorks working more internally and with some big clients of theirs. Wrote a couple of more books. So this will be my sixth book in the Agile space. It’s been quite a ride and I’ve enjoyed a lot of it.

Murray: So let’s go back through those eras. So the first one you called the Wild West, which was in the sixties and seventies. Why did you call it the Wild West? 

Jim: I called it the Wild West because there really wasn’t anything to go by. There was a few books on languages, but there was nothing on design, nothing on requirements, nothing on anything. You had a Cobal manual and you had trial and error, and that was about it.

And then you also had people that you worked with who had been at it a while. So my mentor was a guy called Ed who showed me programming in that environment. His whole office was full of card decks, in boxes with rubber bands and printouts all over the place.

And Ed would run the end of year financial close by pulling a card deck out of his drawer changing some of the cards and running the year end financial statements. There was no backup for Ed, there was no backup for his card decks. If there was a fire in the building, we were screwed. If Ed got sick, we were screwed but that’s the way it was. 

Murray: What about at nasa? Cause you would’ve thought in the sixties and seventies they would’ve had pretty structured methods for the spacecraft operating systems. 

Jim: I really worked on three ships that went out into the Pacific Ocean to track the spacecraft coming back. There was a lot of orbital mechanics and software that was needed to do that. And it was basically experience. There were not many structured methods. People just devised the best way to code. Tom DeMarco told me a story about looking at some code when he was at Bell Labs and they were trying to figure out what made code better. And they said if you can write it without comments, that was a better code. And he actually talked about a program that he had looked at that all the data names were vegetables.

Murray: Code without comments. Great. Yeah, my, first programming job was in a support team that didn’t really use any methodologies, and I remember. As a young developer using PL one that the other more senior programmers would hoard knowledge and wouldn’t help me. it was hard. No debuggers nothing. But on the other hand, I could deploy directly into production whenever I wanted to. 

Jim: Where I was people were fairly helpful for each other. But it was a small group. eight or 10 people. I was the young guy on the floor. I was in my early twenties and everybody else was considerably older. 

Murray: So next, There’s all the structured methods. So I guess we’re talking about waterfall and I remember something called Method One from Accenture. 

Jim: Yeah. Method one was from Accenture and it got started in the 1970s. And got bigger in the 1980s, but the structured methods were really about building software. So there was a data flow diagrams and systems analysis that was popularized by Tom DeMarco. There was data structured Or diagrams that was started by Ken Or. Larry Constantine, developed some of the structured approaches and he did the cohesion and coupling idea.

So we started using pictures to depict designs and requirements and program structure charts. And about the middle of the decade people started combining the structured tools or methods, With some project management stuff that was coming along at that point in time, which was very waterfall oriented. The waterfall article that was written by Walker Royce said, don’t use waterfall. It said use iterative, but there was a waterfall diagram on there. And so people picked up on the waterfall diagram and took it from there. Larry Constantine said he and Ed Horton had used waterfall as a training wheels for new programs, but that it wasn’t really appropriate for big systems. But that sort of got baked into things. And so we ended up with monumental methodologies that had 4, 5, 6, 4 inch binders full of forms and processes and it become very burdensome. And so the solution to that was to have case tools so you could automate all those diagrams and forms. And that was the next error.

Murray: did the structured approach solve problems from the Wild West era? 

Jim: It did. Sometimes I look back and wonder how we did it, but there were actually some very good systems built throughout the era. Part of it was the technology at the time was pretty simple. And so the programs could be fairly simple too in terms of the inputs and the output.

So for example, you graduated from punch cards to character based terminals which basically had no graphics. And the other thing people forget is that we were building systems to support internal users, accounting people, the payroll people, the manufacturing people. It was all internal customers, and so you didn’t have to have the slickest designs to satisfy internal customers. In the 1980s, data flow diagrams were perfect for modeling internal process flow within a business and then modifying that flow to make it more effective and then automating it.

Murray: What were the problems that people were seeing with the structured approach or that you were seeing? 

Jim: There’s two different things. There’s the structured tools and then there’s the waterfall lifecycle. And you would use the tools in various places in the lifecycle. The, one of the problems with the waterfall lifecycle was that it dictated, the organizational structure. So you had requirements design, programming, testing on the waterfall. So you had analysis group, you had a design group, you had a programming group, and you had a testing group. And at that point in time people believed that you could interface those groups through documentation. And so you didn’t have collaboration between those groups. You basically had people throwing big documents over the fence to the next group and I think that was what created the biggest problem.

Murray: If you look at people using the, that waterfall approach today, the common thing is that everything looks good at the beginning, but by the time you get to the end, it’s always way over time, over budget, the quality as poor and lots of stuff’s been descoped in order to, meet some arbitrary deadline. 

Jim: Yeah. One of the things that I used to tell people is that from a management standpoint, agile gave you a very different risk profile than Waterfall. In Waterfall. You went along, and finally at the end the testers had two weeks to test the system. And you found out it didn’t work. So your confidence in that project is high at the beginning and low at the end.

Agile is just the opposite. You have low confidence in the beginning and you admit to that low confidence, and as you build it over time, you get more and more confident. So at the end, you’re very confident about it. But that’s a very different mindset for managers to think about.

Murray: yeah. So then in the nineties people started developing all of these newer approaches and I’ve heard Ken saying that the iterative idea in XP actually came from Scrum. But it sounds like this iterative stuff was around even before Scrum. 

Jim: Oh yeah, it’s been around for quite a long time in a lot of different areas. I think it rose to popularity in the 1990s, but, in the late 1980s, you had spiral model by Barry Beam, and you had an Evolutionary Model by Tom Gilb. Those were the start of the more modern iterative approaches. So with each of those approaches, though you didn’t have the time box nature that you have with XP and Scrum and some of the other Agile approaches. I think that’s an addition that happened in the nineties.

Murray: Yeah, I read RAD by Steve McConnell before I got into Agile. I think that would be part of that period as well. 

Jim: Yeah, I met a lot of people in New Zealand. I met Martin Fowler down there. I met Steve McConnell down there when we were all speaking at a conference. In fact, in the book, I have a picture of a conference that we did in Wellington, and it’s my picture and Martin Feller, Steve McConnell.

Murray: Yeah. Okay. So there’s a group of people in the nineties who were not happy with the structured methods, trying to find different ways, and I guess quite a few of those people were writing books. So how did the Agile movement begin? I understood that there was a lot of discussion going on at object oriented conferences. 

Jim: Yeah, the object-oriented community was always more oriented towards iterative than the business community that was doing Cobal and structured stuff. So there were people in the OO community, people like Ken Beck and Martin Fowler, who were pushing this iterative approach because it was fairly straightforward to do using some of the objects were in language languages.

I became aware of Scrum in the mid nineties. I read a paper that Ken had done. I became aware of Dsdm in Europe. I didn’t know about Allister’s stuff until later. And I met Martin in New Zealand. And we corresponded back and forth a little bit, and I think Martin introduced me to Kent Beck. And then Kent and I exchanged manuscripts in late 1999 and realized that we had some things in common. And this thing of having a meeting started . There was a meeting that proceeded the Agile manifesto meeting that Kent Beck called an Oregon. And this was about a year and a half before the manifesto meeting. And he invited a couple of outsiders. I happened to be one of ’em, so I got to meet the movers and shakers of the XP movement at that meeting. And then Uncle Bob sent out the first email to begin the inviting people to the snowbird meeting.

Murray: Did anybody think that was gonna be an important meeting? 

Jim: No, we all were doing stuff that had been labeled light methodologies. And as Alistair said, he didn’t like being called a lightweight, so we were looking for something that was better than lightweight.

And really the genesis of the meeting was to see if we had some things in common, see what those things were and go from there. There was not an agenda, there was not an objective. We had no idea that the manifesto was gonna come out of it. We just knew that there were some issues with a lot of systems, and that there were a lot of people that were upset with the way things were going. 

When you had a waterfall life cycle when you got down to programming, testing. People were saying, we can automate that. This was a era of business process re-engineering, TQM and the people down at the bottom are just cogs in a wheel they’re replaceable.

This is the kind of thing we were dealing with. This was a quote in the Computer World Magazine in 2001 by a guy who was a c e o of a Indian software company. There is this myth that software development is a creative effort. It relies heavily on individual effort, it is not. It is just very labor intensive mechanical work once the initial definition of specification stage is passed. That’s the kind of mindset that we were dealing with and trying to get past.

Murray: Yeah. I remember reading in the Harvard Business Review around 1990, articles by people saying software development should be outsourced to the lowest cost provider. There’s no competitive advantage to software. 

Jim: Yeah. the other thing that impacted me quite a bit was starting in like 1991, I spent about four or five years teaching Software Quality at Microsoft. And when I first went into Microsoft from my structured background, I kept thinking, these guys are doing everything wrong.

But then I began to think, yeah, but they’re making a hell of a lot of money. And so I started looking at their processes and practices and noticing that they really concentrated on the programming and testing aspect of it. They did requirements and design, but they were really excellent developers. And it really stuck in my head that was the way that the future methodology should be oriented 

Murray: yeah. So then it came together in snowboard to develop the Agile Manifesto. How long did that take? Was it just like an obvious coming together or? Was there much thought gone into it? 

Jim: Not ahead of time. We really didn’t have an objective for the meeting. It was like a kind of a open meeting. We started off and there was no agenda, so we made up an agenda. We spent two days at Snowbird to write the value statement. And we spent the next couple of months online writing the principles. We had started on the principles at Snowbird. We didn’t know it was gonna explode.

We thought some people that would be interested in it, but it wouldn’t be a great big thing. There were a number of people who had some managerial experience, but by and large it was developers and some architects.

But there’s an incident that I wrote up in the book that kind of gives the flavor of what happened there. This is a story that Alice Coburn told, So he and Ron Jeffries were talking to Steve Miller and everybody began wondering, why did Bob invite Steve Miller to this thing? Steve Miller is not a light methodologist. We considered him to be a heavy methodologist. In fact, when he introduced himself on day one, he said, I’m a spy. So he, and Ron and Allison were sitting down talking and Steve was explaining what he was doing in terms of doing diagrams and doing graphics and that kind of stuff.

And then he would generate the code and Alistair and Ron said, we don’t think you can do that. And why would you wanna maintain the code in two places? Why would you wanna maintain the diagrams and maintain the code? Steve said no, I’m gonna maintain the diagram. I’m not gonna maintain the code. I’m gonna maintain the diagram. And Ron and Alistair sat back and thought we don’t think he could do it, but the same as our intent. And so they ended up coming to the conclusion that we were on the same page, whereas we started out thinking we were on very different pages.

And that’s the kind of collaboration and listening that happened time and time again during that meeting where we really tried to figure out where the common ground was. And that’s why we got, 17 iconoclasts and nonconformist to agree on a manifesto. That was a big deal right there.

Murray: How much was it about developing a common name that you could work under? 

Jim: Once we’d gone through and we went through the first day and everybody of the 17 really talked about what they were doing. And then at some point after that we said, we really don’t like the moniker, lightweight, and so we just put a whole bunch of words up on the board. I can’t remember. There was 15 or 20 words and systematically went through an eliminated words and ended up with Agile. And I think everybody really agreed that was a good word.

Shane: So how much of that two days was trying to find a shared language between 17 different people that had started out with different languages. effectively.

Jim: I’ve been in a lot of meetings over a nearly 60 year career, and that meeting was a highlight. You had a bunch of people who we’re type A’s and very independent mind and very iconoclastic. And it just happened. They melded together very well cause we were all facilitators. Somebody would facilitate for a while and then somebody else would pick it up for a while. 

but It was a really good meeting. And I think a lot of it was, there’s a tremendous respect for each other. There weren’t a lot of people out there with us at that point in time.

Murray: How much of it come out of some of that sixties movement values, the libertarian, anarchist empowerment movement. Do you feel like people were inspired by that at all? 

Jim: I don’t know that from having talked to them. I can always say that it probably inspired me a little bit. When I was in college, I went to Atlanta one time to visit my folks. And my dad worked for an engineering firm downtown Atlanta. And so I wanted to go down and see engineers at work. So I got dressed up suit, tie shoes I’m very conservative. Went down to my dad’s office and he freaked out because I had a light yellow shirt on. I didn’t have a white shirt on. It didn’t take much to be a nonconformist in those days.

Murray: And if you’re working for ibm, it was the blue suit and the red tie

Jim: That’s right. 

Murray: But there is a strong feeling of, we want empowered teams, trust the team, trust the professionals. It’s all about small teams of highly skilled people doing good work in a collaborative way, like the people who are at the manifesto itself. That’s what I see as being an important thread through it.

Jim: Yeah. I think that’s an important thread. I believe that there’s still a place for good leadership. And in fact, in my first book, adaptive Software Development, I devote a lot of time to something I call leadership collaboration, management as opposed to command control. And I think a lot of that got lost. A lot of the agile people rejected project managers and project management. They said, eh, we don’t even need any management. We don’t need any leadership. We just have self organizing teams. I told Ken one time, get rid of the chicken and pigs. He didn’t do it.

Murray: Yeah. There was, people at that time saying when the revolution comes, the managers will be first against the wall. 

Jim: Yeah. One of the things I wanna write up is something about management and leadership and the fact that it got a bum wrap. Jergen Apple, wrote a book called Management 3.0, and he’s one of the people I interviewed in the book for his new unfixed model. He was saying that he got into writing that book because he was a manager and he brought in Scrum and he didn’t like the ideas that they had around management. 

I was in Germany, this was like 2004, 2005, and it was a big electronics company. And I was supposed to give a presentation to a group of about 15 software development managers from all over Europe. And so I made a little presentation to this group and then one of the managers stood up and he said, our scrum consultant says that we can’t ask how much it’s gonna cost, or how long it’s gonna take. What do you think about that? And I think said, I think that’s utter bullshit.

Murray: Yeah, I agree. 

Jim: You don’t run companies like that.

Murray: No, you can’t. That would be. economic suicide. You should be able to say, it’s worth this much to me and I need something that achieves this goal in this time. 

Jim: People say, agile doesn’t work if you’ve got a fixed time deadline. I happen to have been working on projects with people that had fixed deadlines that couldn’t move, and we did it just fine. 

Murray: Yeah. Me too. I find the best approach with Agile is what I would call a fixed capability, fixed team, fixed time, fixed cost, clear goals, but, we’ll flex the scope in order to achieve what we can in that time. 

All right, so we’ve talked about how leadership is missing, also this idea of larger organizations is missing with Agile cause it just focuses on the team, which then led to a lot of issues with scaling later on. So the idea of the organization apart from the software development team, was missing, I think.

Jim: Yes, it was. I have always looked at Agile as a generative methodology or a generative approach as opposed to a prescriptive approach. And a prescriptive approach says RUP in the old days, rational, unified process SAFE today. A prescriptive approach says, let’s tell you everything that we need to do in infinite detail. A generative approach says, let’s start with what we absolutely need. And then as we need other things, we’ll go out and bring them in. We don’t wanna start with all that stuff, particularly if we’re not a huge, big organization. 

One of the things I found with RUP many years ago is it was huge. And you tell people, let’s scale it down, and they couldn’t do it because you can take any piece and say, would this be useful? Yeah, that might be useful, but this piece might be useful. And pretty soon you’re not taking anything away. So it’s much better to have a generative approach where , as you need stuff, you add it than a prescriptive approach where you try to do everything at once.

Murray: We really like patent libraries. Like an ala carte menu, you can pick what you want rather than having something very prescriptive. And then you can have a lot of patents in your patent library. You don’t have to use them all. 

Jim: Yeah. The first pattern I ever used was updating a master file with a transaction file in the days where you had tape units and the last, record on the tape had to be all nines as an in file indicator.

Murray: Yeah, so there is an idea within the Agile manifesto that the business will tell us what to build. And a lot of people are now saying actually we should have empowered product teams where the team does discovery with customers and works out what to build as well. 

Jim: I can see how that might be an interpretation that somebody would make. The thing is that we weren’t really a product management, product development, group of people. And I think that’s something that we really lost or that needed to be added . If you look at how these projects went in the early years, they were basically teams of developers. It was hard to get the testing people involved and it was hard to get product management involved. And that grew over time.

In the mid nineties the internet caused us to move from internal systems for internal clients to external systems for external clients. And that was a huge change. You didn’t have to have a very sophisticated interface design when you were dealing with internal people, but when you are dealing with external people and they can click on some other place to get that information then you’re dealing with a whole different type of customer. And I think that began to draw in more product people into the development process. 

Shane: Yeah, we’re building something for internal audience, they’re mandated to use it. So they have no alternative. They have to live with it, whether it’s good or bad. Meant that we didn’t need the skills or the domain expertise of building products that people wanted to buy when they had a choice. With the introduction of internet more people can build things more often and provide alternatives changed the set of skills that we need because we need to understand what people want to buy, what they’re going to use, how we build it, how we make sure it works, and therefore, 

the breadth of skills is broader than the lens of just software. The question that I’ve always had is, was there actually a conversation to say this could be used anywhere? 

Jim: Yeah. The manifesto can be interpreted in different ways, and I’ve interpreted the only deliverable is, working software I can blow that up into a product. In fact, if you look at the value statements that I used in my project management book, they’re just a different interpretation of those. And to me that’s okay to do something like that.

Murray: So if you were gonna do it again today, what would you do different. 

Jim: It can’t be done today. And what I mean by it can’t be done is when we did the original agile manifesto, the market was tiny. There was not many people doing it. There was, the group of us and a few other people that were doing things that were similar. Today, you’ve got this huge marketplace. Everybody’s doing agile. Whether they’re doing it right or not is a question we could have, but there’s no way that you could pull together a group of 10 or 15 or 20 or 25 people today and write a manifesto that would satisfy all of those different marketplaces. And so I would suggest to people that rather than try to redo the manifesto itself is that they look at their target audience and write a manifesto for that audience. As opposed to trying to write something for this whole huge marketplace that we’ve gotten now. Cause it’s just a different time and place.

Murray: yeah. That’s a good point. 

Jim: If I was gonna do it differently today, I wouldn’t do it. 

Murray: Well, lemme put it another way, is there anything in there that you regret or that was much harder for people to adopt than you thought it was? 

Jim: The thing that seems to have been harder than I, than we anticipated is the damn word over.

Murray: Yeah. 

Jim: And we meant it as a way to make decisions, to help people make decisions. People and their interactions over process of tools, did that say that process and tools was unimportant? No, they’re very important, but ultimately it’s the people themselves that are gonna do it. One of the interesting things coming out of the manifesto is that you had people and their interactions are the most important point coming from a bunch of techies.

Murray: So why did that y’all take off. 

Jim: I think it has to do directly with the values, not the process and tools, not the methodology. But the values that were incorporated into the manifesto spoke to a lot of people at a gut level.

Murray: Yeah. 

Jim: And that’s the kind of thing as we move forward and try to go to Agile two or Agile three, or whatever we’re gonna call it. What we really have to do is to say, what is it that we want to keep and what is it that we want to change? Is there a core set of ideas that we wanna keep , as we go forward with this thing?

Shane: When you talk to people, there is really a close coupling of the idea of Agile and Scrum being one and the same rather than the manifesto being the core and these different things you can do to work that way? 

Jim: Yeah. I think that has happened. I think it’s happened because Scrum has, essentially took over the marketplace. When the manifesto was written, XP was really in the lead. And once Scrum started in on the training and the certification that really took off and established Scrum as the heavyweight in the field.

And with something that big, some people are gonna do it right, some people are gonna do it wrong. I know a number of people Mike Cohn, for example, Who’s been an advocate of Scrum and XP and the technical practices and a really good way of doing Scrum. And there’s some other people that are extending Scrum into some much better places. I think there’s still a lot of people out there that are doing what I call prescriptive agility, which is an oxymoron/

Murray: There’s a lot of water Scrum fall out 

Jim: Yeah. You shouldn’t have to put an adjective in front of agility. But I think you do in these cases. And so in the book, I really talk about methods, methodology, and mindset. And all through, as we’re talking about agile, as we’re talking about waterfall, as we’re talking about other kinds of things, I really try to tie those three things in and talk about them. And ultimately it’s the mindset, the agility mindset. 

I started trying to answer the question for myself what is agility? What does that mindset mean? And so what I did is establish what I thought were the actions of an agile leader. Because I think it takes adventurous nonconformance to be agile. And if you can’t show me somebody that’s out in front with the aero sticking in their back, then you’re probably not gonna have a good agile implementation in your organization. You gotta be willing to take chances.

Murray: I think that the core idea of Agile is that there’s always a lot of uncertainty about what the problem is and what the solution is in anything to do with software development. And the whole approach is designed to deal successfully with that uncertainty. 

Jim: I developed something first project management book called The Exploration Factor. And it takes into account the volatility of requirements and the experience that you have on the technology. So if you have a project in which you’ve never done this technology before and you’ve got a high level of uncertainty, that’s a 10. And if you’ve, technology that you’ve used a good bit before and the requirements are fairly fixed, that’s a one. And what a lot of people don’t do is they don’t look at how difficult their project is and how uncertain it is. If I’ve got a one project, hell use waterfall, it’ll work just fine. But if you’ve got a 10, you better not try Waterfall on it cuz it won’t work with the dam because of the uncertainty.

Shane: And also it can differ. Within the same piece of work, you could be building something and some of the stuff has a high level of certainty, Because you’ve done something like that before, or the team have, you think you know what you’re gonna, you need, and half of it could be highly uncertain, And so you have to adapt the way you work within that one product.

Jim: In the chapter on courageous executives, I talk about sciex up in Canada and the product development manager, Ken Del Call, says exactly that thing , which is not only do you have an expiration factor for the product, you have one for various pieces of the product. You better be looking at those things that are tens and working on those early and often. So you get that risk out of the way.

Murray: So that’s the next decade you talk about in your book 2000 to 2010, which is the courageous executive. So that’s really when it takes off. 

Jim: Yeah. It’s when people began coming to me and to other people and saying, we wanna implement Agile throughout our IT organization. They weren’t yet thinking about the whole business. They were thinking about the it. So I would hear from a VP of engineering at a software company.

I did one that had about 500 engineers and was completely waterfall, completely big methodology. Josh Kerievsky and I worked on that one together. I worked on a number of these moderate size, up to about 500 to a thousand people in development. I was at a conference one time talking to a VP of engineering for a Chinese company and I said you’re doing the Agile?

He said, yeah, we’re doing a few Agile projects. I said how many Agile projects are you doing? He said, oh, we’re doing three or four this year. And I said how many you wanna do next year? He said, 200. I thought, not likely.

Murray: It worked very well with the first teams, didn’t it? I read Ken’s book in 2003, and I’d read some of Alistair stuff before that. But really when that manifesto came out for me it was like a beacon. I knew there was big problems with structured software methods, but I didn’t know what the answer was. And this pointed me in the right direction. And then I started reading the books from all the various authors. So it was a way of, bringing everybody together under one umbrella for me that I could then explore. And then when I went and implemented it, it worked great on small teams. It solved a lot of problems for me as a project manager. But when I look at it, what’s been happening since then, it’s been sporadic in its success. I see people have tried, it works really well with some teams and then, they try and roll it out. It doesn’t work. They roll it back. People are bringing in structured methods over the top again. So sometimes it works, sometimes it doesn’t. Do you have any feel for why that is?

Jim: Yes. One of the last things I say in the book is that individuals and interactions and collaboration is the root of all success, and people in their interactions are also the root of all failure. I started thinking about how often politics gets into big projects. It’s about people and their interactions. And the larger the project, the more people you got involved the more, it’s hard to nail down where the problem is. It looks like it’s a technical problem, but it’s actually a people problem. How many companies are around like Apple? They’ve been very successful at innovation over a long number of years. Everybody can see what they do and they talk about what they do, but you can’t duplicate it. And I think it’s the same thing with software development. 

Salesforce took on Agile very early and dominated a marketplace because they switched to Agile. They started completely redoing their software because it was buggy and they couldn’t scale it. And so they redid it and that whole company became agile. So it was a company strategic decision and they made it happen. There are a lot of companies that pay lip service to it and that’s what they get out it. Over the years, I have seen every single methodology succeed and every single methodology fail.

Murray: Yeah. I think organizations are basically bureaucracies and agility does take, quite a different way of thinking and organizing and leading. Which you might call, flat organizations. 

Jim: I like , Jeurgen Apello’s UNFIX model. It’s a structure that’s modifiable. It’s like putting a set of Legos together. And I really like that. I was at a big meeting in ThoughtWorks in about 2015 and the question was how do we address the scaling issue? You had safe and you had a couple of other approaches to scaling and clients were asking what are our thoughts about scaling? And we realized that we were addressing the wrong problem. It wasn’t scaling processes and organization that was a problem. It was scaling innovation. Safe says I’m Safe. Our product was Edge. We’re edgy.

Murray: I gotta ask you about Safe, Jim. Do you think that safe is agile? 

Jim: I’m sure that some people have implemented Safe and they’ve been agile but the nature of the beast when I first started looking at their big diagram, it’s like, what in the hell are they doing? It will help scale, but I’m afraid it’s gonna help scale the bureaucracy I don’t think you can scale innovation with SAFE. A digital transformation of a company needs something that is edgy, that depends on emergence, that’s not all that prescriptive that’s more adaptive. And I don’t see those things coming outta SAFE. 

Murray: The other problem we see in our industry at the moment is this big switch towards non-technical life coaches becoming agile coaches. What do you think about that?

Jim: So tell me a little bit more about that one. Cause that. was completely new to me. 

Murray: Yeah The executive coaching community got heavily into agile coaching cause of the money available. And we’ve got this whole group of people now saying that technical skills get in the way. Any experience of software development gets in the way because then you might tell the team what the solution is and really what you need to do is empower the team to understand that they have a problem so they can come up with the solution themselves.

Jim: So are you telling me that I can’t contribute to the technical issues on my project? That’s bullshit. I may have some really good ideas about what would make this product better. Are you telling me I can’t talk to the product owner about those kinds of things? I think you have to take the people and the roles and meld them together and look at what kinds of things can this person do versus what can that person do and adjust. One of the things in Scrum that I hate is this idea is one head to chop off. The product owner, you chop his head off if it doesn’t work out. Well, you chop the team’s head off. You don’t chop the product owner head off. 

Murray: Yeah. Responsibility should be shared by the team. Not just held by one person. 

Jim: So the coaching that you were just talking about, is that more at an executive level?

Murray: No, we are getting agile coaches hired now across, say four teams that have never been part of a successful agile team or a software team, or have any technical expertise whatsoever. They’ve come out of HR quite often, and they’re being hired in preference to people who have been part of an Agile team or a software team.

Jim: I think that’s a sure way to fail. 

Murray: Yeah, I think so too. 

When I look at my experience with where Agile’s worked well and what hasn’t it seems to me that it’s worked best with groups of people who are quite competent and skilled. And it hasn’t worked well with a group of people who are low skilled working for another company in a developing country. That’s where it’s been most difficult. So I wonder how much of this is actually dependent on having skilled professional people in the team that you’re working with locally or closely?

Jim: You’ve seen two trends here. One of the trends is for people to use Scrum or XP prescriptively. A non-thinking approach. You just follow the process. And I think that’s an issue. If you have people who have got some experience and have some knowledge about the area, they’re the ones who can think through, I can use this, or I could change this process to meet our particular needs. And I can adjust things. It’s the people who can’t adjust things and be adaptive that cause the problem. And those are the ones that follow the prescriptions. So I think it has something to do with experience, but I think it also has to do with the mindset of being willing and able to change things. And so it may be what organizations have imposed on the team as well as the team itself. I think if you have people who are, have less experience, that there can be a place for them on an agile team. But if you’ve got everybody that’s inexperienced it’s not gonna work.

I was in India a number of years ago with a company that was a CMM level four, trying to go to a CMM level five, a capability maturity model. And, they had this technical review. It went perfectly. You could look at all the documentation and it looked great. Everybody signed off on it. All the things were there in the documentation, but it was a terrible technical design because nobody on that review team had any experience with the technology. So they went through the process, but they had no skill.

Shane: Do you think that it has got to that stage where it is the late majority trying to adopt it and therefore it’s done, its dash, and something new is needed or do we just need to hold ourselves accountable to what the manifesto was about. Where are we in the life cycle? 

Jim: I think if, Agile was static, it would be time, to change to something else. But if you look at what Agile has generated, agile has generated Kanban, DevOps and continuous integration, that’s something that grew out of the agile movement, that enhances the agile movement.

So I think there are those kinds of things that have happened over time, and I see some of that happening today like with Agile 2. Alistair Cockburn has a heart of agile. There are a couple of other things that are going on, like that. And I see that rejuvenating agile. So the question is what do we keep and what do we change? And I’m not opposed to, some changes that need to be done. I don’t think it needs a rewriting of the manifesto. I think it needs a reconnecting with some of these advances in software development, whether they’re agile or not, doesn’t make any difference. We can end up calling this thing Excalibur for all I care. It has some pieces that are agile, some pieces that are something else. You’ve got new tools and methods and so you build a new methodology around that because software development is a combination of what’s being asked for in the business world, what the technology is, and what the software development processes are. And as the technology changes, as what we’re asking for changes, then the methodologies the way we approach things have to change too. You wanna move towards agility. That’s the big thing that you wanna continue to do.

Shane: You started off talking about a bunch of software developers that got together and created the Agile Manifesto. And you’ve just closed out saying, if we’re gonna regenerate, it’s still bound by software. That’s where it came from and that’s where it’s bound to. 

Jim: Not in my head. It’s not. I can expand that into agile leadership and executive management and I’ve written about that several different times. I think that those things just enhance agile. 

Murray: So what’s coming up next in our space? What’s after agile?

Jim: One of the things I thought about when I wrote the book was, am I gonna try to predict the future? We’re in a period of punctuated equilibrium, where everything is changing. It’s not survival of the fittest it’s more survival of the luckiest.

One of the reasons I wrote the book was to help people prepare for the future by understanding more of the history. An example of that is this no-code, low-code thing. That happened in 1990s. It was called End User Computing. And it had some of the same problems and some of the same issues. So are we gonna learn something from that? Possibly not. I was talking to a 30 year old colleague not long ago, and the only thing that she had ever done in school or in work was agile. She had no conception of what went on before. And so that’s what I hope to do with this book is bring some of that history so that people can look at how we move forward and take some of the stuff that’s relevant and good out of the agile movement and take it to the next level.

Murray: We better go to summaries Shane.

Shane: Excellent. All right. You started off talking about structured methods and organizations, has been hierarchal and fixed mindset versus the agile movement, which is for me all around growth mindset. 

Why did we need to change the way we worked? And that was because our business owners was telling us it costs too much, takes too long and you don’t gimme what I want. That’s the core problem we’re trying to solve. We’re not implementing Agile as a methodology, we’re trying to fix those three things. 

I like the idea that you talk about Wild West, where we had languages, but we had no design. A whole lot of things we’re missing. And so it was understanding a language combined with your experience to make the thing work was the era that we were in. 

Then we moved on to structured methods where we had tools and we had process and life cycle, and we bound those together. That actually it was the waterfall diagram that got it traction. Because it’s a simple diagram to understand. 

And you talked about the Agile manifesto, that when you were part of the team that wrote it, you didn’t realize how successful it was gonna be. I always wonder if you had a realized how successful it was gonna be, whether it actually would’ve been done in two days.

And then we went on to, roots of Agile and where it came from and then the agile era and that idea of a courageous executive. Near the end. You talked about, this idea of agile mindset because it’s always been hard when people say what’s agile? And the answer is, it’s not Scrum. What is it? What’s a mindset? You know, when you see it. Are you behaving in an agile way or not? 

I like the idea that, one of the problems we had with Waterfall is it dictated our organization design. So it was a consequence of implementing a process. We structured our teams in a certain way and then we ended up with conversations via documents, and we know that’s not the best way to collaborate as a team or share information. 

And with Waterfall we start with what we think is confidence until we run outta time. And then we end up with a mass of uncertainty and a lack of confidence cause we didn’t deliver. Whereas with Agile, we start off consciously saying we have a lack of confidence and we’re aiming to work towards certainty. 

We’ve had some, a lot of people on the, podcast from the product domain, and there’s people that are research focused; so they talk about UX and research and going out and understanding the problem first and quantifying that actually is a problem. And then figuring out how you might solve it. And the idea of like minimal viable product, lean startup where you think you know what the problem is and you think you know what the solution is and you’re gonna go and experiment and test that.

And that research led process makes me feel like Waterfall, because you think you can identify a problem and have certainty that is the problem and therefore you can find a solution. And now it’s just about validating that the solution works. Whereas I think half the time we don’t even know what the problem is and we guess the wrong problem even after research. 

And the last one is this idea of internal versus external, that it was done in a time where the audience was focused around using what you gave them versus now we have choice.

So yeah, it’s been great, lots of really good takeaways for me and as always, more things for me to go read and go and learn up on. Murray. What do you got? 

Murray: Yeah. Look, I think the core issue we are dealing with Agile is all about uncertainty. The problem and the solution that we are trying to solve by building software products is far more uncertain than management thinks it is or the team thinks it is. If we gave ourselves a rating of three on the uncertainty factor, we’re highly likely to find out by the end we should have given it a rating of six or seven.

There’s a lot of belief that we are certain at the beginning, but we’re actually not, and I don’t think I’ve ever seen a software project that ended up the same way that ,we thought it would be at the beginning. So that’s the fundamental problem. Lots of uncertainty about the problem solution. Agile is a great way to deal with that uncertainty.

It’s really all about a set of values and principles. It’s not frameworks or methods. I found it very helpful. I think the new things are, heart of agile, continuous discovery, continuous delivery with empowered product teams.

Scaling it for me is not safe. Safe is just the new bureaucracy. Safe is the new heavyweight methodology. It’s no surprise it came out of the RUP world. To me, scaling is best done through pattern libraries using Unfix. There’s also scrum patent libraries, empowered teams, and agile leadership, which is all about servant leadership and not the typical autocratic way of working. Interestingly, there’s a lot of parallels between Agile leadership and mission command in the Army. And one of the key things they’ve found is that the amount of empowerment you can trust people with depends on their competency and skill. So the greater the competency, the more empowerment leaders should be giving their team. And the lower the competency, the more frameworks they need to put around them. So that’s what I got out of it. Jim, we should point people towards your new book from Wild West to Agile. 

Jim: Yes, Adventures in Software Development, evolution and Revolution. It’s due to be released the End of May. 

Murray: Great. Looking forward to it. and it, do you have a blog where you write and people can find you.

Jim: Yeah. Jim highsmith.com is my website, but I’m doing a lot of writing on LinkedIn these days.

Murray: All right. It’s been wonderful having you on. 

Jim: Well, thank you. I’ve enjoyed it.