Ways of Working with Scott Ambler

Shane Gibson and Scott Ambler discuss the essence of constructing bespoke ways of working with data teams.

They discuss:

• Scott Ambler’s Background: Scott talks about his extensive experience in IT, starting in the mid-1980s. His journey includes working internationally, starting as a programmer, moving into data architecture, and pioneering in data warehousing.

• Shift from Heavy Processes to Agile: Scott’s career saw a transition from heavy process methodologies like CMM to Agile approaches, emphasising flexibility and adaptation over rigid structures.

• Agile in Data and Modeling: Scott discusses the evolution of Agile in data spaces and modeling, highlighting the radical changes from traditional practices, like the shift to using Post-it notes and whiteboards for planning.

• The Importance of Context in Agile Practices: Scott emphasises the necessity of understanding the unique context of each team and project, debunking the concept of “best practices” as universally applicable solutions.

• Empowering Teams to Choose Their WoW (Ways of Working): The conversation underscores the importance of teams choosing their own WoW, based on their specific context and challenges, rather than following a prescribed methodology.

• Critique of Methodology and Framework Overdependence: Both Shane and Scott critique the tendency of organisations to rigidly adhere to specific methodologies or frameworks, often leading to inefficiencies and a lack of true agility.

• Continuous Improvement and Measurement in Agile: Highlighting the challenges in measuring the effectiveness of Agile practices and the need for continuous improvement within teams and processes.

• Agile and Data Quality: Scott shares insights on applying Agile principles to data quality, emphasising the need for techniques that address specific challenges unique to data projects.

• Scaling Agile Practices: Strategies for scaling Agile practices across multiple teams, stressing the importance of adapting practices to fit different team contexts and needs.

Read along you will

Shane: Welcome to the Agile Data Podcast. I’m Shane Gibson.

Scott: And I’m Scott Ambler.

Shane: Hey Scott, thank you for coming on the show. I am mega excited about this one. I’ve been a big fan of your work for many years. And what are we going to talk about today is this idea of ways of working. How can teams build their own ways of working? Why should they do it? How do we enable them?

What steps should they start with? But before we rip into that, can you just give the audience a bit of background on your career in this world of Agile and data and ways of working?

Scott: Yeah, definitely. So I’ve been in the IT space, since the mid to late 1980s, working in various positions in various industries in various countries. And I’m actually based in Toronto, Canada, but I was international, an international traveler. For many years, and I guess I still am, I started out as a programmer, but very quickly moved into the data space became a baby data architect.

I guess you would say a phenomenally junior data architect at a very large financial institution. And pretty much did their first data warehouse long before data warehousing was even a concept, and learned a lot, I actually was the holder of most of the corporate data at one point, at least the metadata about the corporate data.

At one point, and the reason why the data architecture group had brought me into the fold was basically because I had more information about what was actually going on than they did, and people knew it. Then I moved into the object oriented space and human computer interaction, like user experience now, and and modeling did some work in the mid 1990s in the 

cMM space a very heavy process at the time. I actually wrote the first CMM level five compliant process. If you instantiated it fully for object oriented development. And basically once I finished that work off was. Pretty much disgusted by CMM and moved on from that and went to the other end of the spectrum and started focusing on Agile stuff UML and Agile, Agile modeling, Agile data, and it took on a lot of the early 2000s, took on a lot of the not so sexy topics in the Agile space like modeling documentation in particular and it was interesting now we take it for granted, but back in the day, it was radical, like doing things on post it notes and on whiteboards.

And although, to be fair, people were doing it, but nobody was talking about it. And if you pick up modeling books from the late 90s, early 2000s, there isn’t a single sketch in them. There isn’t except for the books that I wrote and you’ve got to take those out of play on that one, I keynoted at a academic modeling conference once and basically ripped into them for that, right?

So I had their conference proceedings in my hand. And it was really good really solid papers. But I basically, I was asking the audience do you guys sketch and use Post it notes, raise your hand and they’re all raising their hands, and, raise your hand if you’ve got a paper in this conference proceedings, and it was really thick. Good portion of the audience had published. And I said, so how many of you include sketches in your papers? Everybody’s hand went down and I knew it would because I flipped through the proceedings and I said okay So this is a problem .

You’re not writing about what you actually do in practice you’re writing about this theory, which is why pretty much everybody ignores you And so that was a yeah Wasn’t the most popular guy, but anyways so it opened up the modeling space in the Agile world.

And so all the things we’re doing now were really was radical 20 some odd years ago. Also did a lot of work in the data space. , did some basic fundamental stuff there long before, other people can including yourself and all the good work that you’ve been doing there.

And arguably it was too early, it was a good 10 to 15 years too early because the data folks, just couldn’t accept the fact that Agile was real and here to stay and they didn’t want to hear it. And the Agile folks didn’t care. Their idea of data knowledge is, Oh it’s a database.

We’ll just encapsulate access to it. And that’s all I need to do. So no worries. And that would be it, right? And then they’d go off and make, horrendous mistakes. So I did a lot of work there. But then when I joined IBM in 2006, I was their chief methodologist for IT within IBM Rational.

And there, myself and a few others, including Mark Lyons, Developed Disponagil Delivery, which then evolved into Disponagil and then eventually into PMI’s Disponagil Toolkit. And that’s where we really made it clear that teams should choose their own ways of working and choose their own.

Wow. And we described how to do it and different techniques, but it’s all about choice and understanding your context and giving up in this concept of best practices. Like best, the term best practices is it’s a wonderful marketing term, but it’s an absolute lie and it’s what people want to hear, but it’s observably not true.

The best you can do is. Let’s do the best we can. For the context, for the situation that we face. What makes sense for me in my situation would be a spectacularly bad idea for you and your situation. So we have to help people instead of telling them what to do and say, this is .

The official practices of methodology X. And if you do these, then everything will work out for you. Instead you need to help people understand here’s the things you need to think about, here’s the considerations and the issues that you’re going to have to deal with, and here’s options to deal with those issues.

So choose the right thing for you. Choose the right. Ingredient off the shelf to make dinner for yourself tonight, right? And your dinner tonight will be different than what you made last night and the night before, and that’s okay. But you need the right ingredients and use them in the right way for, whatever mood your family is in tonight for for dinner.

Shane: I’m a baby in the Agile data space. I’ve only been doing it for I’m probably almost hitting 10 years now. But I remember when I first started this journey, I definitely came out of a best practice methodology driven mindset. That’s what I got taught when I was learning in my early career.

And then I got introduced to this idea of Agile, and I was like, you’re a bunch of weirdo hippies. 

Scott: Pretty much.

Shane: No way, that’s just chaos, how the hell can you do that? And I remember there were three seminal things for me that helped my journey. One was a series of books by Ralph Hughes, which was giving me the insight of Agile terminology and Agile things in data warehousing language.

I could see the blending of those two. There was yours around what I… Now I realize it was a patent library, a bunch of, good practices that have some context and I could pick and mix things that fitted for my customers. And the third one was Lawrence Core’s stuff around interactive modeling with customers using stickies.

And I delivered a session for a bunch of data engineers . Just lately, and I was showing them how they can use the 7ws and Beam to elicit data requirements early and lightly. And one of the questions that came back from one of the engineers was why don’t you just use the ERDs? And for me that was, yeah you try and do ly relationship diagram modeling with a stakeholder and see what kind of feedback you get.

’cause you lose them. I think in the data domain, we still haven’t adopted that idea of light engagement with our stakeholders, with a language they understand to get what we need to do the next step. I’m with you that in the data world, we’ve got better. But I don’t think we’re anywhere near the level of sophistication in this space that we should be.

Scott: , exactly. Recently I’ve developed and started delivering a workshop in Agile data warehousing. And I used to do that a few years ago, but then had to stop when I went to PMI because they just weren’t interested in the data stuff at all.

And, Orner enterprise architecture stuff or a bunch of other things they, they bought and then just. Put on the shelf, basically and rightfully so it wasn’t their business. But one of the key messages in that workshop is that on a data project if you’re doing a data warehouse or BI solution, you want to be usage driven, not data driven.

The data is actually a secondary issue and that blows the minds of the data people. And this data focus is one of the primary causes of their high failure rate is because it’s just too narrow and it doesn’t really reflect. What people want to do, right? If you look at Lawrence Kors work, he talks about data stories and in my stuff, I talk about question stories and they’re very similar concepts but they’re both usage driven requirements.

And that’s, what’s critical. If you don’t know what questions, , how people are going to use the data, what questions they’re trying to answer.

I don’t care what your data structures are. I don’t care what you’re storing. I don’t care what , data technologies you’re using. It’s all nonsense. And this is why, if you look at a very classic problem, the data warehousing space is the first release is a loser.

And it’s because you build it, you release it. And then the stakeholders come along and say, this is not what I want. It’s, 90 percent there, but there’s 10 percent that’s desperately missing. And I can’t do anything with this because this 10 percent is missing all that sort of stuff.

It’s not until your second or third release that you get it right. Or at least you start getting closer to getting it right. And it doesn’t have to be that way. The final problem is that they wasted, sometimes wasted years doing data modeling and nobody cares.

And they don’t even keep an update usually. In practice, even they don’t care sometimes. If nobody’s using your models, why would you, why did you bother creating them to begin with?

Shane: What we’re finding now, especially with self service ETL, the whole practice of an analytics engineer, this idea that you can build code that transforms data without modeling. And self service has massive value we allow more people to do the work that used to be done by specialists.

But when we go to that model, that self service paradigm, we often lose the rigor or the skill. And we’re definitely seeing that in the data modeling space. And again, cloud technology has given us some massive benefits in the data space. So we can now deploy platforms and build them and cobble them together much faster than we used to be able to do in the old days of, installing Oracle databases and ODI on servers.

But effectively what we’ve done is we’ve taken a screwdriver that delivered a hole in a screw nobody wanted. And then we used a power drill to deliver it faster. But it’s still, a screw in the wrong place. We just delivered it faster. So we’ve got to get into that model of early feedback.

How can we build something small, get some feedback from our stakeholders? Because at that stage we’re going to learn what we did wrong, what we misinterpreted. Because, we don’t have a shared language with them as much as we hoped we did. So just thinking about that and just thinking about this idea of way of working, because that’s the whole focus of the podcast today.

How would you describe way of working? How would you describe Wow if you had to explain it to somebody from the beginning?

Scott: So your way of working is your process, , this is how we go about doing our work as a team, as an individual, as an organization as the case may be, and the fundamental thing is you want to do what’s right for you and do the best that you can in the situation that you face and always try to get better what that tells you is you need to understand what context you’re in. You also need to understand what you’re trying to achieve. In Disponagel, we talked about having, process goals, basically what are we trying to do? And what do we need to be thinking about?

So there’s a little bit of process knowledge there once you understand what you’re trying to achieve, like I’m trying to, Understand the requirements, for this data warehouse that we’re building, for example, then you need to understand what are my options?

How could I go about understanding these requirements? Maybe I could do ARDs, maybe I could do user stories or data stories or question stories or use cases or this or this or this. Maybe you could do screen sketches or some, random data sketch that I’ve never seen before, right?

But somebody else has figured it out. And the idea is that you’ve got choices, but which one of those techniques is right for you in your situation based on what you’re trying to achieve, what your skills are, what your culture is what your preferences are. And then how do you choose which technique is right for you, particularly when you probably only know one or two of the 15 that I just listed.

And . Number seven was probably your better option, but if you don’t know about it and if you don’t know what the trade offs are, how the heck can you make a coherent decision to experiment with it and see if it’s right for you? Because just because technique number seven sounds like a good idea doesn’t mean you’re going to be able to pull it off.

I hope you can, stuff happens. This is the idea, right? You’ve got to have enough knowledge. Or somebody on your team has to have enough knowledge. Or you have to have some sort of a toolkit or, some sort of magical AI tool perhaps that can give you advice as to here are your options.

You’ll hear your choices between them. Choose wisely, right? So it’s the Indiana Jones thing.

Shane: , I think for me that the idea of a toolkit, that kind of what resonates and just lately I’ve been on a whole analogy of food in the data space. I don’t know. I always seem to go back to food as the analogy and I liken it to a buffet. We have a bunch of choices that we can put on our plate.

We may have team members that are allergic to egg, so some of those food choices, they’re not available to us as a team, given our context. And so what we’re doing is we’re picking from this pattern library, we’re picking from this buffet of choices, and we’re Fitting the ones that work for us, given our context and solve our problems.

And then we’re saying, okay, that looks like it’s going to be a good, nice meal. Let’s try it, right? Let’s experiment with it. And that’s what a lot of teams don’t do. I see is they pick things, they draw them, write them down. They therefore become immutable. This is our way of working. And they don’t go to the next step, which is actually, we’re just guessing, we think that pattern is going to solve that problem.

But we need to experiment with it. We need to test it. We need to do a retrospective on it that says, did it? Yes, no. If the answer’s no, don’t carry on doing it. Find something else, right? If the answer’s yes, then lock it in for now. But at some stage, you’re probably going to change it again. Is that what you found?

Is that, your way of working is incremental, isn’t it? It’s a toolkit that you’re constantly adding to and removing and iterating as all Agilis should.

Scott: Exactly. And actually in DA, we use the the buffet metaphor for quite a while. And we also used cooking your own dinner as well. Because you could. Compare that to, going to McDonald’s and getting the Big Mac meal deal. And yeah, you could eat that every night, but it gets boring pretty quickly and not everybody in the family wants it.

The interesting thing about the buffet analogy is that if you actually observe a family at a buffet. , everybody gets fed, but they’ll all go and choose their own food and they’re all much happier than whatever it is you were going to cook them instead. And sometimes, you’ll pick something, what the, what is this one or what this is and you’ll pick it and it’s great.

Or you’ll pick it and, Ooh, that’s not for me. And you’ll leave it on your plate and it gets taken away a little bit of waste. , you had an experiment and some experiments work well and some don’t. So it’s interesting. Now it’s a little bit more work for the people.

It’s not just, I go to the table, sit down and eat what’s in front of me. It’s, I’ve got to, wander all the way over to the buffet, make my own choices, wander back to my table, eat, and maybe rinse and repeat a few times and do that. So it’s a little bit more work, but I get a better meal I get a meal that’s going to meet my needs and everybody eats differently.

And that’s it. Absolutely important to observe because every team is unique. Every person is unique. Every team is unique. If you want to be effective, you really need to , allow your teams to have unique ways of working. Just like you allow your children to have, pick your own food to buffet.

Now, with my daughter, sometimes I’ll say, you can’t eat, you can’t just go to the dessert. Part of the buffet. You gotta have some vegetables. You gotta have some, real food first, and then you can have some dessert. So you gotta give them some guidance.

A little bit of governance is required, but you gotta be smart about that, but you still you let the kids pick their own food and, some experiments don’t work out so well, , and that’s okay.

Shane: If we look at, the data space, like I said, I think we’ve lost the art or the skill of modeling to a degree. At the moment, it’s starting to come back we always go through waves of decentralization, self service, and then moving back to more managed, governed skills based, and then we go back out again.

One thing I observed is data teams are really good at looking at processes from a data platform point of view. The idea of collecting data, profiling cleansing conforming consuming it, they understand that process very well. That’s their specialty within that domain.

But if we look at business process or team process, that whole idea of process engineering that we have from many years ago, most data teams don’t seem to understand that, and they seem to struggle to take this idea of they can map their data process and they can see nodes and links, and they can see the first step and the second step, and they can see the.

And, the entry into that step, and they can see the exit, and they can see what the dependency for the next step is. But when you ask them to do that in their way of working who’s going to do what? And then what are they going to produce? And then what do they do next? They seem to struggle.

And have you found that? Have you found that skill of understanding team process and business process and lean process is lost at the moment, in the data space or just generally?

Scott: I think it’s a general issue. I think the data crowd tends to struggle with it more just because, All these communities are self selecting, right? So the people who move into the data community tend to be really good at structural thinking and, understanding data and all this sort of stuff.

And they might be a bit weak on UI and they might be a bit weak on, business processes and stuff like that. So then as soon as you ask them to step out of that, safe space for them then yeah, it becomes a bit of a mess. but frankly, it is a hard skill. It’s a level of abstraction that a lot of people aren’t used.

It’s really hard because there’s so many, it depends situations, right? You really can’t describe it well because, it’s the same old thing, right? You ask people to describe what they do. Tell me about your day and they’ll miss 50 things and they’ll, describe things that they didn’t do and that aren’t needed.

And they’ll describe things that, only they care about. And if it was to disappear tomorrow, nobody would notice things like that. It’s a hard problem. And it’s going to change anyways, imagine you had to define your process for tomorrow.

Here’s what I’m going to do tomorrow, that plan will last maybe 10 or 15 minutes and something’s going to happen, . Somebody asks you to do something, you get a call, something breaks. Earlier today I was coding and for some reason, Python wasn’t seeing my GPU anymore.

Which is problematic cause I’m doing some AI stuff. So I really want access to my GPU. So now I’m going to be fooling around for a couple of hours, probably trying to reinstall CUDA, and all that good sort of stuff. It is what, my day is shot. Cause that’s anywhere between five minutes and five hours.

I don’t know yet, but so stuff happens your situation changes and you’re going to change up the way that you work and that’s okay.

Shane: If we look at the Agile market at least we’ve gone, to the death by frameworks death by methodologies in my view. And, naturally as humans, it feels comfortable it feels comfortable that rather than, Building our own toolkit, having to do the cognitive load to go, what are my problems?

What can I find? How do I test it? How do I see if it’s going to solve that problem in the way I work? It’s much easier to get a picture of somebody else’s drawing with a bunch of steps and then follow it. Have you seen that? Have you seen that, the market’s moved away from this kind of craft, something that’s fit for purpose and customized for you and your team to adopt a standard methodology or framework yet again?

Scott: Yeah. So I think there, there is a lot of comfort in being told what to do. And even with intellectual workers, they still want to, generally want to be told what to do. And. Organizations also want to tell their people what to do. The leadership wants to be leading and be managing and doing their jobs.

But the fundamental challenge is that these methods aren’t fit for purpose for whatever your purpose is, right? You get, some things like Scrum, which are right, there’s almost nothing to it. And it’s so wishy washy. That, you can force fit it into anything, but then it doesn’t really give you much advice as a result, other than to run a few meetings when it gets down to it.

And then at the other end of the spectrum, you get the safes of the world and they’re overly defined and, they’re prescriptive with, one way of doing things and , they talk a good game and they give you a few options now. So it’s gotten better, but it might work for some people, but for the most part, it’s probably not going to fit well for you.

So then you’re still going to have to tailor it. And it’s so complex that you don’t know how to do it. And you don’t know what to take away. You don’t know what to add. So you’re still sort out a lot, but you’ve got a pretty poster and you’ve spent a lot of money on training.

so things generally don’t go well. It really gets down to, you need to know what you’re doing and , that’s a hard message, but you need to be skilled, you need to have the skills to do your job and, I would say this to executives, I would say this to management, I would say this to the people in the trenches, you need to have the skills to do your job, you need to understand what your job is and how it fits in.

And you need to understand, whatever the purpose is of your, of the team that you’re working in now. So if you’re on a data warehousing team, you’d better have a idea about data warehousing. If you’re on an AI team, you better have an idea about AI and be pretty good at it. And , this I think is missing.

These methods don’t go into the details they try to be, situation agnostic ? So yeah, you can do Scrum, but it tells you nothing about data warehousing. Tells you nothing about building apps. It tells you nothing about marketing and like all these other domains where you’re applying it into.

So you’re left to deal with that anyways. In that case. People being led down the garden path. It’s the art of the possible. You can do anything you want, which is absolutely true. But then you’re stuck figuring it out on your own anyways.

And then there’s not a lot of advice there other than we’ll hire some expensive consultants who generally also don’t know what your domain is and what you’re trying to achieve, so this is one of the reasons why the shine has come off of agile coaches lately. Is because there was just this overwhelming number of people who weren’t qualified.

And, they took a two day course and maybe, we’re on a project or two now they’re pitching themselves as a coach. How offensive is that? And now they’re wondering why can’t I find a job? Because, the entire community is, not really qualified.

And people have finally figured that one out. So it’s problematic, so you’re still stuck anyway. So yeah, you get pitched this story that, we can tell you what to do, but then it’s not really, you don’t really get it. And to go back to the food analogy, it’s like telling people, yeah, you can cook your own meal and here’s the book, the joy of cooking.

Good luck. No, you need to have some. When I talk about fricassee, fricasseeing something what the heck is that? I don’t, it’s, I don’t know. You better have some basic cooking skills to, to get the job done, right? Where are you going to start?

Shane: For me, I differentiate leadership from coaching. , my opinion is a strong opinion is coaches should have played on the field. They should understand the domain and then they spend all their time not telling somebody how to do it. But coaching them how to do it, right? There’s a difference between tilling and coaching.

But if you don’t understand what they’re doing, then you don’t have the context. So that’s what a coach does, and then a leader gets out of the way. So for me, a leader doesn’t have to have domain knowledge, but they have to empower the team who does, to do the work and lead them and remove the barriers and let them get on with the thing that they’re experts at.

Help them, get rid of the roadblocks that are in their way. So if we look at that. Let’s say we’re lucky enough that we have a team and that team’s being suddenly empowered to build their own way of working. That team’s in the data domain and, they have some skills in that domain around data and all that good stuff, but they have limited skills around their business process or the way they work.

And you’re coming in to help them. Where do they start I know there’s no one way to start for every team, but generally, where would you tell a team to start working first in terms of building their own way of working?

Scott: I generally start to wherever they are, right? So if they’re already, in progress, then I start looking around. , first I want to observe and, get a handle on what’s actually going on rather than what they’re telling me is going on.

But I also want to start. Trying to understand what are their main challenges right now? And, where’s the pain and , let’s deal with the immediate pain and hopefully try to get some, quick wins as well.

It really does depend on what situation they’re in and what they’re trying to achieve. And is it realistic, so I’ve been on a lot of teams where, they’ve got this completely unrealistic date to hit or this, completely unrealistic, scope for, their budget or, whatever’s going on.

And they’ve basically given up , or they’re frustrated and they should be giving up. I haven’t quite figured out to give up yet, , I try to help them, understand where their main problems are and then, start knocking them off one at a time.

As we’re doing that, it becomes a coaching thing of, helping them understand that they’ve got options and asking leading questions and pointing them in the right direction sometimes too. And sometimes as a coach, you do have to tell them what to do.

If you’re a hockey coach, and the team milling about on the rink and the other teams have to score on it. He’s get your butts down here. Take that guy out. If they’re nowhere near that level of figuring out that they need to be, protecting the goalie or something.

Sometimes you do have to, give them more advice than . You probably would like to, but often it’s really all about, , guiding them into helping them make their own decisions. And then, standing back , and sometimes, you pretty much know it’s going to be a bad, this is not going to work out well, but you know what, let’s see what happens.

Going back to the my daughter go, a couple of years ago, she went to the buffet and grabbed a big plate full of shrimp, because she likes shrimp. Like, how much shrimp can you really eat? And so we found that out. It was about half a plate, which was pretty impressive, but still not the full plate.

So I ended up eating a little more shrimp than I probably wanted to that night. , it’s a bad idea. But, let’s see how it plays out.

Shane: Oh yeah, and definitely a number of times I’ve said to teams, in the early days, I said, that’s never going to work. I’ve never seen that work. And then, experience taught me to say, I haven’t seen that work before. I’ve seen teams try it. Here’s the problems they hit, but give it a go.

Cause you know, you get surprised. And for me, it is about that context. So I frame it in my head now about a team starting up from scratch will focus on certain patterns. We focus on the team building the ways of working, the team charter those team forming skills.

If you go in and you’re helping a team that’s already down the track, then you’re starting to look at where the problems are. And what needs to be changed next to, to help their way of working. And then if you’re at the stage where they’re scaling, they’re trying to break out to other teams, and that’s a different problem, a different set of context.

And for me, actually, I’ve got to the stage where I don’t actually care which problem they want to solve, because it’s not solving the problem that we’re there to help them with. It’s helping them understand that it’s their way of working, it’s their process, and they need to be able to define and iterate it.

So it’s teaching them the skills or giving them the knowledge of how to iterate their way of working, regardless of which problem they’re solving next.

Scott: Exactly. Teaching them how to learn.

Shane: Yeah, and so that’s that success, right? Is when that team can iterate their own way of working. They don’t need us anymore. That’s when we’ve done our job because they’re just going to keep building better practices.

And so that’s one of the things in Choose Your Whale that I loved was this idea that you always focus on the goal. First so what’s the goal we have to achieve and then what’s the problem that’s stopping us getting to that goal and then which pattern could we adopt that may solve that problem to achieve that goal?

And I really like that way of thinking about it, which is, that end state and then experimentation to get us there with our practice, With our way of working. So that was something that, I’d forgotten actually until I read it again. It was like, okay, , forgot that little puppy.

So this idea of a hybrid , bunch of good ideas, because again, if we talk about way of working I’ve followed what you’ve said for years in terms of there’s a bunch of patterns out there. There’s patterns in Agile, there’s patterns in Data, there’s patterns in Product, there’s patterns in Lean, there’s patterns all over the place that we can adopt.

And they have value in given certain contexts. But what we end up with then is we end up with teams working differently. Because that’s our goal is they’re going to work given their context. But from an organizational point of view, it seems scary, hold on, don’t we want to standardize all the teams?

We can go to that horrible Spotify model problem, which is, Spotify had a strategy and the strategy was people build their own way of working. Somebody documented it once and everybody goes, Oh, that’s how Spotify works. And, I’ve talked to a few people from there and they’re going bollocks, actually.

That was just one visualization of one state at one particular point in time. But as organizations, we always want to go actually it’s just adopt that and put that onto every team. Have you seen that? This idea of standardizations across things seems to be more important than enablement and getting the job done.

Scott: exactly. And this is what CMM and CMMI were all about back in the day, having repeatable processes. , the politest thing I can say is that’s just ignorant nonsense. It’s observably not true. Every person is unique. Every team is unique. Everybody’s in a unique situation. And all of that stuff is changing constantly.

So having one way of working across all teams, now you might magically be in a situation, if you were a consulting organization and , all your salespeople were selling was the exact same basic type of problem or same type of project to all your customers, yes, maybe you could have a reasonably.

Repeal process across all those teams, but you’d still have teams of different people and the clients would still be different. It would all fall apart instantaneously, so say all the planets were to align and you could actually. Somehow find yourself in a situation where every single team following the same process makes sense.

Within a few hours the situation will have changed for at least one of those teams and then suddenly they’ve got to start doing something else or continue to apply the same. Inappropriate process for them, which, and if they’re smart people, what they start doing is they start doing the right thing anyways, and then they start creating the artifacts that you insist on to make it look as if they’re following the old process

so you fake out the process and then that’s just all useless overhead. That’s probably not what you’re looking for. And the interesting thing is often the people who are pretending to govern are unable to dissect that’s happening because they’re almost always looking at the artifacts as opposed to the actual actions of the people that are going on that, the people that are actually doing the work.

This is just, it’s no good it’s what people want. Like they want a simple solution, but the world is too complex. And I think what’s happened is, I work with organizations and everybody’s overwhelmed. The complexity has gotten too high. They have all this legacy is all this technical debt and all this data technical debt.

And the customers are asking for a wide range of stuff. The technologies are changing constantly. You can’t get caught up on the solutions at all anymore. Everybody’s just overwhelmed. including the leadership the leaders don’t know what’s going on either.

They’re trying to get a handle on what’s happening and it’s just too much for them. And yet they’re trying to make decisions and they just can’t it’s far too complex and it’s not what people want to hear. Somebody comes along with this snake oil, follow this methodology for doing this type of project , or for, doing it across your entire organization.

And it sounds good. Like the story is always good, particularly when you don’t really know what you’re listening to. And you’re overwhelmed and you desperately want a solution and somebody comes along with this semi complex thing that sort of makes sense to you, and then they’re slick.

They got a slick salesperson or, slick story. And it sounds good. So you adopt it and a couple of years later, you find out it didn’t really work out for you. It’s a disaster. So you go on to the next snake oil salesman and this is what these organizations are doing constantly.

They just lurch to the next hopeful solution , building up all this technical debt and cultural debt as they go and not really getting the job done. The only way you can be successful these days is you’ve got to step back and you have to accept complexity.

You’ve got to embrace the situation that you’re in and realize that you need to get a handle on it. You need people that understand what their jobs are and what to do and how to do it and be willing to learn. Because when everything’s constantly changing, you’ve got to be constantly learning and changing yourself.

And this is hard. And it’s what it’s all about.

Shane: yeah, skills and knowledge, that’s what why we get paid so much. And often I see, facades like you talked about. One of the things about methodologies or standardization is it makes people comfortable because they see something that looks familiar. 

As humans, we look for patterns and we go, seen it before, feel comfortable about it. But we see this facade behavior, so an example I had was, worked with a team, the team started rocking it in an organization. They were amazing. Organization, as often happens, they see this little aneba, this little virus start to happen that’s working differently, and they try and kill it.

And the team, had some good, umbrella or shit cover. So they managed to survive they had a leader that protected them from the political norm. And the team survived. So then the organization changes and goes, Oh, okay. This thing looks like it’s working.

We should adopt it. But what the organization did, because it was a PMO, project management office led organization they brought in a BA to write the methodology on this agile way of working and that person at no time talked to the team. They didn’t talk to the team, they didn’t absorb it, and then this document came out, said here’s our agile way of working, and the team went what do we need to do?

We need to change the way we work to that, because that’s not what we’re doing. And we tried that bit, didn’t work for us. We haven’t tried that bit, maybe that’s interesting. But this organization standardization seems to be the way we’re taught, right? Because we’re taught to chicken farm, we’re at school, we’re taught to follow a set of steps and do it that way.

But if we talk about that waste, that idea of somebody that’s spent six months of their life writing a document for their agile way of working that’s never going to be adopted. Another example of waste is, the team are operating in a certain way and then a project manager has to distill it into a Gantt chart for the PMO because that’s the standard way we report across the organization.

If we focus on documentation within the team, their way of working, what level of documentation does a team need to do to document their way of working, do they document everything they do? Do they document nothing? What have you seen in terms of the right balance of describing the way they work, while they’re building it?

Scott: Once again, it really does depend on your situation. So for example, if you’re in a regulatory environment where, legally you have to have a defined way of working, then you’re going to probably do a little more writing than if you weren’t in that situation. But it really depends on what your team needs and what your what new people would need.

So as people come into the team, how do you onboard them? Some organizations will, here’s your process binder, read this and then, try to stay awake while you’re doing it. And then at the end of the week, we’ll come back and, talk to you and maybe next week you can start doing some work.

At the other end of the spectrum you just start and you start pairing with people and you learn through osmosis, how things work. I’ve seen processes defined as simply as a. A single page on the wall, piece of paper on the wall for a co located team. Here’s our agreement.

Here’s how we intend to treat each other. Here’s our rights and responsibilities. Basically I’ve seen 50 page documents, several hundred page documents that, nobody ever reads. Even the new people, they get 10 pages in and , it’s all over by that point.

So the problem with documentation, people probably aren’t going to read it. Don’t write the documentation if you don’t intend to keep it updated. And so if your way of working is constantly changing, then you’d be constantly changing , whatever documentation you’ve got about that.

Same old rules, lighter is better than heavier when it comes to documentation. Keep it as concise as possible. Just barely good enough type thinking. I would not invest a lot of time in writing process documentation, unless I’m in a regulatory situation that requires it.

And always read the regulations. Don’t let the bureaucrats Read the regulations and tell you what they say. You need to read the regulations yourself because, what the regs say and what you’re told the regs say. are often very different.

Shane: Yeah, so it whispers, right? As multiple people read and talk, we change the language and we change the intent. Yeah, I’m with you. I think it’s the right level and that’s hard. One of the patterns I’ve seen that was particularly successful was This idea that the documentation often was targeted at new team members,

the problem we’re trying to solve is new person coming into the team, we have this unique way of working, how do we describe it as distinctly as possible to get them started? And so the documentation focused around that. And the new team members were always tasked with this question that, if you came on board again, what’s the one thing you wish had been documented or described for you that would have made your life easier?

The, oh, I wish I hadn’t known that. Write that one. And then what’s the three things you didn’t bother to read? And then over time, when you get enough ticks in that box, you decommission that piece of content because people are telling you it has no value. And then, like you said, the level of documentation needs to align to your cadence of change.

If you’re constantly redefining the way you work, you’ve got that high cadence of change, then less documentation because it’s going to get out of date. And out of date documentation is far worse than none. Because people trust it. They read it and go, oh, that’s the gospel. And if it’s out of date, they don’t know, versus if it doesn’t exist, they’re going to go ask a team member, how do we do this, 

and they’re going to get the latest version. So I’m with you on that and so that idea of continuous improvement, we get taught to do continuous improvement in our code, continuous improvement in the way we deploy, in the way we build, and in our data, and that whole continuous improvement It’s something that’s baked into us now when we work, but very rarely do our teams get taught to do continuous improvement in the way they work.

Now, if we look at Scrum, that’s where the retro came from, the retrospective, that idea that we’re going to force a ceremony, where we’re going to force the team to say what went well, what didn’t, and what are we going to change? Because we know by bringing in circuit breakers, by bringing forcing functions, people will learn that behavior.

It’s quite brutal. And it is a form of waste if we think about the amount of time we waste with that team meeting to do that retro versus if they just retrospectively changed the way they worked every day. But we know those forcing functions were important. But outside Scrum we don’t tend to see that, Continuous improvement of the way we work, of our processes being endemic, do we?

Or have I got it wrong? Cause Lean’s all about process review and continuous improvement, isn’t it? 

Scott: I think it really depends. So I would argue that I don’t think we really see a lot of improvement in Scrum. It would have changed, like Scrum hasn’t changed all that much, since the mid 1990s, it’s, it’s evolved a bit. But considering how many people are doing it and how long it’s been around it’s really stagnant.

And I think it shows it and people have been saying this for years and then nothing happens be just, because of the culture of the scrum community. It’s more about holding a retro, right? So I think the challenge with forcing people to have a retro every week, every two weeks, whatever your sprint length is that it becomes so regular that it becomes nothing.

And, are you really improving? Are you really having a retro? Are, are you really identifying what we should. And then fix it’s that get off your butt and deal with the issue or issues. That’s where it all tends to fall apart. Cause otherwise, these scrum teams would evolve away from scrum within a few months.

I guarantee that because every time , I’ve been coaching a team and we actually did retros and I held their feet to the fire, by putting measures in place so I’m a big fan of measured improvement. If you identify, hey, we need to fix X, whatever X happens to be.

Okay, fine. What are we going to do to fix X? Let’s do this. Okay, fine. What, how are we going to measure? How are we going to know that we fixed X? Oh, I don’t know. We need to figure that one out, right? What improvement do we need to see? And how are we going to measure that over the next, month or two?

However long, we think it’s going to take to fix X. And then, do we see improvement? And then, and it’s not just declaring success. Say it takes you two months to address whatever this X thing is. I want to keep some measures in place to make sure we’re not slipping. Because what’ll happen is, it’s the shiny new problem syndrome.

You think you fix one thing and you really barely fixed it. And then you go on to the next thing and you forget about X and then your behavior slips and you start doing not X because now you’re working on Y.

And I think this is the challenge with a lot of teams. They really do stay stagnant and it’s that lack of discipline around improvement that really gets them into trouble. , like I said, feel free to quote me on this. If Scrum teams were really doing process improvement, they wouldn’t be doing Scrum anymore.

Shane: I find scrum’s a good place to start. It gives us some really light patterns that we can explain, we can coach, teams can adopt, they can learn. And then they should get the hell out of scrum, they should go why are we doing two week iterations? Why don’t we just make it continuous?

Why are we doing retrospectives as a meeting at the end? Why can’t we do it every day? Why are we doing daily stand ups? Why can’t we just continuously talk? But without those forcing functions, without those experiments to show a pattern of learning, And then measure whether we’ve got the success. We don’t do any of those things.

So I find Scrum incredibly useful because it’s a set of valuable patterns but , it’s not the answer, it’s just part of the journey. And that measuring work versus measuring the way we work that’s incredibly difficult. So if we measuring work is hard enough, so we can say yes, we can see people are busy,

they’re doing work. Everybody’s busy, tasks are being done. But are those tasks moving any of our dials? In the data world, did that data provide any action, any outcome, any value to the organization? Hell, was that data accurate? We don’t even measure that half the time because we don’t know what accuracy actually looks like.

In the product world, in software we can say, yes, we put a widget on the page and people can type into it. But does that change the behavior of our customers? Did that customer behavior change actually deliver us more or less value to the organization? Those measurement things are really hard.

And when we apply them to the way we work, to our processes, it’s hard as well, if I don’t do this, is that a positive outcome or a negative outcome for the team and the organization? So measurement seems to be, as humans, one of the hardest problems for us to solve. Have you found that in that, we talk about experimentation or patterns.

Okay, if I apply this, given this context, it should make things better. But measuring what better looks is incredibly difficult. Have you found that?

Scott: Yeah, definitely. So the challenge there is, are you really measuring what’s important? So for example, in the data world it gets back to, the data is a secondary issue, it’s really the usage of the data, the questions that I’m helping you to answer, in case of a data warehouse if we’re helping people answer questions 

and make better decisions. That’s what I want to be measuring, right? Did we actually help you answer the questions that we promised that we would help you with? And more importantly, have you moved on , to better questions? Because over time, things are going to change

and your priorities will change. That’s what I really want to be measuring. So in the case of way of working if it’s a technical practice, like some DevOps thing, then yeah, I can instrument the tooling and I can measure, how much are we at, how often are we doing, are we actually integrating and how often are we running the tests and stuff like that.

So there’s some easy things to do there, but. That’s usually not the case. So it gets back to, the, the problem with X. So what was X, what are we trying to improve with X? X would be, I want to improve data quality so then I should be measuring data quality, not the fact that I’ve adopted some new technique or some new tool that improves accuracy, Have we actually improved what’s important to us? So measure. What you’re trying to improve or what you’re trying to adopt rather than the techniques that are Helping you hopefully to improve and so it becomes fuzzy at that point right because quality improved, You know, I want to improve quality and I changed this I changed this way of working to help improve quality.

And quality went up, but we could have also changed two or three other things in flight as well which might’ve also had an impact on quality. We don’t quite know, cause some other technique that we’ve been experimenting was really the source that had a side effect of improving data quality and we just didn’t clue in

so anyways, it was hard to measure there’s always going to be some qualitative decisions going on, we wanted to improve this, the metrics are showing we improved this. It was probably because we changed this way of working.

Okay. We’ll, we’ll continue working that way until it doesn’t make any sense. 

Shane: And it’s that macro versus micro. From a data point of view, the way I frame it now is I want to understand the business question the stakeholder wants to answer. So it normally starts with how many, how much, how long. Then the next thing I want to understand is if I give you the data or the information to answer that question, what action are you going to take?

I want to understand all the customers are going to churn. Alright, once I give you that, what are you going to do? What action? Oh, okay, I’m going to go and, do a save offer. Okay if you take that action, what outcome are you expecting? Okay if I do a save offer, I’m expecting less people to churn.

Okay. And if that outcome was successful, what’s the value to the organization? Oh, okay. I expect that, less people churning, lifetime value is going to go up by 5%. Now that’s great. I’ve got a nice. Cause and effect model there from the old balance scorecard days. But it doesn’t mean that, we can’t have gone and done a dumb campaign that caused churn.

It can’t mean that we took a stance politically in the market that people didn’t like and they churned. So there’s a bunch of other macro, things that may affect that value, but at least we’ve thought about it. Now, what I haven’t done is taken that pattern and applied it to the way of working.

The best I get is, what thing are we doing, what change are we making, and what action do we expect to impact? We’ve got a problem we’ve delivered some stuff to stakeholders on a regular basis and they turn back and say, that’s not what we wanted. Yeah, so typical data problem.

OK let’s change the way we engage and let’s do something slightly earlier, slightly lighter, using a different language and see if , that impacts it. So our action is to engage with them earlier and get, better requirements effectively. And then I can see that

I can see we have engaged. I can see that they felt that was a good conversation. I haven’t flowed that through now to say what’s the actual outcome of that action? And then what’s the value and how do I prove it? So I’m still struggling with that measurement part of changes to the way we work.

Still work to be done, I think.

Scott: Oh, for sure. Yeah. And just get used to being frustrated., there’s a lot of qualitative stuff going on there. And there’s so many interconnections that it’s really hard to tease out. We changed this one thing, how much of an effect really happened?

And there’s things that are going on that you’re not measuring at all. So you’ll adopt a new way of working because you want to improve quality, but a side effect of that new way of working is you’re now ticking off. One third of your customers, customer satisfaction is going down while quality is going up.

But you’re not measuring customer satisfaction, or if you are you’re not relating it back to this new way of working because you never thought about it. And this is , one of the things I’ve continued doing that is when I describe a technique.

I describe both the advantages and disadvantages because there’s always trade offs. And I think this is where a lot of people writing about methodologies and, ways of working really missed the boat. And this is something I learned from Kent Beck, like 25 years ago, when he described XP, he was very clear.

This is when this will work. This is why it works. This is when I suspect it’s not going to work and here’s why. And I extended that thinking , to the detailed practice level of, here’s the advantage and disadvantage of this practice and this is very true in patterns, 

if you look at the way patterns or, particularly process patterns are described , there’s the context, the original context in which this pattern will work. And then when you apply it, here’s the new context that you’re in. If you successfully apply this pattern, there’s good things and bad things about this new context.

And this is just the way the world works. When I’m describing choosing your way of working, I describe it in terms of risk, what risks are you willing to live with? What risks do you want to pay down? What risks are you willing to increase and learn to live with?

Because you will, there’s no perfect solution here. And this blows people’s minds. And this is why I’m always concerned about this term best practice because people spin it as well. This is the best way of working and this is truly awesome. Yeah. In the context in which you came up with it.

But what were the downsides of that? Were you even able to detect the downsides or are you so inexperienced in your own technique that you haven’t seen it fail yet? Because it will fail. Guaranteed it will fail at some point for some reason. And if you’re not aware of that, you might not want to be pitching it.

Honest about it and say, you know what, I haven’t tried it in this situation. I suspect it’s not going to work, but I don’t know for sure. Cause I haven’t tried it, if anybody tries it, let me know. And fair enough best you can do, don’t pitch anything as this is perfect and it’s going to work for everybody in every situation because it’s not, and it’s what people imply.

They won’t blatantly come out and say it that way, but they imply it. All these people want to sell you certification in certain things. They’re implying it for sure.

Shane: It’s about describing anti patterns as well. So if you know you’ve applied this pattern in a certain context and it didn’t work, describe that anti pattern, describe the behavior of why it didn’t work. It doesn’t mean it won’t work next time in that exact same context. Because as you said, there’s other contexts around.

There’s the organizational mindset. There’s the way the organization is structured. There’s the budget constraints. There’s a whole lot of other things that impact it. But describe that anti pattern. An example for me would be… Our stakeholders are frustrated because they can’t get any work into the queue.

natural pattern, introduce the concierge. Introduce somebody who’s going to go and talk to them early, gather something, put it in the queue, engage with them, don’t just say you have to wait. That’s a great pattern. The anti pattern, now what you’ve done is set a promise that you’re not going to deliver because if you haven’t changed any other way of working, their work’s not going to get done any faster.

But you’ve set up an anti pattern where you’ve had a conversation, you’ve set an expectation, and now you’re going to disappoint them even more. So that’s your trade off. Your trade off is, do you want them to feel like they’ve had some input, and then manage the expectation? Or do you want them to feel like, actually, there’s just no way they can get their work done?

Or do you want to find another pattern, which actually fixes the problem, which is the flow of work and the speed you’re doing it. Describing those anti patterns is important, but what I find is describing a pattern in the written word is incredibly difficult because I can do it waving, I’m waving my hands now,

I can wave my hands when I describe a pattern. I can look at people’s faces. I can listen to their questions and I can change the context of the way I describe it. And yeah, with a pattern, there is that standard. Pattern for a pattern, which is, here’s the thing, here’s the context of the thing, here’s how you apply the thing, here’s some risks or anti patterns where it’s dangerous for that thing

you’re trying to distill all this knowledge so somebody else can pick it up and apply it. But there’s so much context around it that it’s hard to do in the written word. Do you find that? Do you find distilling patterns down to things that, you’ve written it, I can read it and apply it, that is an incredibly hard process.

Scott: Yeah, it is. And and this is why, if you look at the work of the patterns they can take years to write up a pattern language, and it’s common to take years to write up a pattern, language and get it right. The first few iterations of what you think is a great pattern language tend to be total garbage.

And that’s okay. But you learn and they shepherd you through and it’s tends to be a multi-year process. But you end up with some pretty good results. And the challenge is it’s also so slow, but then you end up with these multi page patterns.

If you read the Gang of Four book from, the mid 1990s there’s what, 25 patterns or 30 patterns in that book, and it was a 400 page book or 500 page book. You end up with these 10, 15 page patterns. Now, there’s source code, so that, blows out the size, these patterns get pretty big pretty quickly if you’re doing it right.

In DA. We had 800 practices put into context in the toolkit. So you can’t have a five page description for 1, 800 things, right? That just gets too huge, too quickly. So we would describe each practice in a paragraph or two. The trade offs were bullet pointed and then we would link to an, oh, if you’re, if you’re interested in learning more about use cases.

After reading a couple of paragraphs about use cases. Go read this book or go to this website or go watch this video. And so we pointed the details and leave it up to you if you were, you’re interested in that. But because we were focused on just putting things in context and helping you decide and that’s it.

Not teaching the actual practices. And we still had ways of putting the practices into context of, to help find, like earlier, we’re talking about how do you find this stuff? You’ve got to be able to traverse these things called process goal diagrams to get in the right context.

So in this context. Here are some good options, and here’s what each option means, and here’s the trade offs, and then, go read a book if you have to learn more. And that’s hard, right? But it’s the way it is. Reflects the complexity that we actually face.

It gets back to this challenge that people are overwhelmed with the complexity. Even when you simplify it to the point we did they still get overwhelmed with the simplified version of the complexity. And yet. You ask them and it’s, Oh, it’s so big.

And it’s so hard. Yeah. Okay. What would you take away? Oh, I don’t know. Oh, but now I think about it. You’re missing these five things. Yeah. We know we’re missing those five things. We haven’t gotten to them yet. But you just told me to me, it needs to be more complicated, not less complicated.

I think is the challenge that people just overwhelmed and they don’t even want to. Put the effort in to understand what they’re trying to achieve. So I’m in school right now. I hinted at this and I went back, I’m working on a master’s in AI 

so a wide range of students and I’m having a lot of fun and, we’re all learning together and it’s all good. A few of my fellow students there’s huge range, right? And a few of them are, a couple of years in, so they’re younger. It’s interesting, they’re in a master’s program and a few of them are anti education and it’s I’ve pulled myself up on my bootstraps and I learned how to program on my own, which is great. I’m sure they’re great programmers, but I can safely say there’s this.

Very long list of things that they don’t understand and they probably don’t even know they need to understand it, their architecture skills are probably horrendous and many other things and, testing skills are probably not there. And that’s, they’re learning,

But this is the problem, right? Here they are in this very complex domain and learning the hard way that maybe there is a little bit more to it than what they think. So a bit of a wake up call, and I’m watching another gentleman right now a friend of mine and he’s writing a book and he’s researching , this stuff and he’s constantly surprised.

Oh, this isn’t the way I understood it. And he’s learning so much, right? He’s learning a lot and which is great. And I’m thinking, you know what, like all the stuff that you’re 30 or 40 years, That was second year university for me. So it’s great that you’re finally catching up, maybe if you been a little more open minded to get, to education you wouldn’t be learning the basics now.

At least stumbling into the basics now. I don’t know how I got off on this tangent, I think people need to invest in their education and finding a good university program or college program and whatever it is that you want to do is hard.

Believe me, but it is worth your while to invest . in understanding the fundamentals of what your job is, because the fundamentals don’t shift all that much. Look at the data community, the fundamentals really haven’t changed that much. There’s some new toys and, and yeah, no SQL databases, but you know what, object databases, blah, blah, blah, right?

And there’ll be some, new toy to play with in 10 years. But the fundamentals are still the fundamentals and yeah, there’s more volume and it’s changing faster. But you know what? In the late eighties, I was dealing with what I thought was a lot of volume and it was a lot of volume at the time.

There’s a couple more zeros tacked on to the end now, but so what? The technology can handle it. My laptop I suspect I have more processing power on my laptop than I had on the mainframe in the late eighties.

My workstation is more powerful. My real workstation is below my laptop.

Shane: It’s about trade offs decisions. I was talking to somebody the other day and they, were talking about their codes, for their warehouse, their transformation code. And, the question I had was how many blobs of code do you have?

And they said something like 10. And I was like, okay, that doesn’t gel, right? The pattern of , small number of blobs of code for the size of your organization, the amount of data sources you’ve got, and the time you’ve been running. So my natural reaction was, okay, there’s lots of lines in that code

yeah, they’re massive. Because they’re going from source to consume, and the code is the model. And I’m like, okay, there’s some trade off decisions you’ve made, you’re not modeling, you’re not decomposing, there’s a whole lot of things that are anti patterns in that. But there’s also value in doing it as one big blob of code.

Was that a conscious decision of you or your team, that was my question, because I don’t care that you’ve done a pattern that I don’t agree with. I just care that you’ve done it for conscious decisions and you can change those decisions when it happens. And the other thing that’s complex about describing patterns for me is patterns are patterns of patterns, 

an example I use is when I describe change data capture. The ability to see that something changed, and I want to recognize that it’s changed. Our English language says CDC has changed data capture, there’s a pattern. But we also use that exact same term for log mining to say we’re going to go and hit a database log to see that change.

We use a slightly different… Language around delta detection. If we’re going to compare two tables, states and say what changed, but they’re all change data capture patterns. So now I’ve got this one word that I’m using multiple times for multiple different patterns. And so for me, when you’re describing a pattern, it’s make sure you put something in the front.

change log, change data capture. Delta, change data captioning, describe why they’re different. Because we often have patterns that are made up of other patterns. Like your friend, I’m a failed author at the moment. And what’s been interesting for me is there’s been a bunch of patterns that I’ve used with teams for many years and found teams very successful.

And so I coach teams on those patterns on a regular basis. When I started writing, I could write what the pattern was, the anti patterns, the successful application of it. But I had no idea where it came from. What was the origin of that? And the one the other day I was just updating my article on T Skills or T Shapes and I was reading your one where you go, don’t call it that, call it Specialist Generous,

And then you had the T Shape diagram that I use as a template with all the teams I coach, and I was like, oh! Did that come from Scott or did that come from somebody else and again, it’s the origin story of… Where do these patterns start? Where do they get blended? And I’m intrigued by that,

where’s the origin of some of these things? I just heard on the radio today that the first zombie movie was can’t remember, it the Dead, but it was the other one in 1968, right? That was the first time somebody actually made a movie about zombies.

Scott: the living dead. Yeah, George Romero.

Shane: but there’s an origin story, right?

We know that was the first video that was published because people probably made some at home. 

Scott: Actually, it wasn’t. I’m a zombie guy. It wasn’t. It was the first zombie movie of that genre. Yeah, so Romero kicked off what we now… Pretty much, all the zombie movies are basically the same story or the, the same basic pattern. But before Romero’s stuff, there was zombies, zombie movies where there was like one or two zombies, and there was like a witch doctor would bring somebody back to life and you could argue that Frankenstein was a zombie thing too, but not really.

But but yeah, no, there was some zombie movie stuff before that but Romero kicked off the… The billion dollar, and I don’t know what it is, but it’s like huge zombie industry now.

Shane: So yeah, there we go. Agile data, ways of working. We’ve got some loves. Zombies, movies are one of mine and Zombieland still is my favorite comedic go to, sit down and relax movie. Watch it time and time again., so patterns the patterns are patterns. You go back to the origin story.

I’m really interested in that. So there’s a lot of complexity in there. And the last complexity I just wanted to cover before we close out is scaling your WoW. Let’s say we start off with a team, small team, they build their own WoW. We’ve now said, okay, we’re not going to PMO it, right?

We’re not going to write it up as the methodology and mandate. That’s how teams work. But we want to scale the learning. From that team and those patterns and we want to share the patterns of being successful with the context and the anti patterns so other teams can pick it up. How do you typically describe to an organization when they want to scale their way of working to other teams that are doing similar work

we’re not going to take the way of working from a data team and try and apply it. Regularly with a software development team, these shared patterns, let’s say we’ve got a data team and we’re just scaling out to two or three more squads, pods, teams, whatever you want to call them.

What do you recommend to customers? How do they do that?

Scott: There’s a couple of patterns or common strategies to do that. So for example, I’m having, communities of practice or communities of excellence, like coaching teams and tribes or not tribes guilds, and I can’t remember the other term from Spotify. It’s basically another team or person outside of your team whose job or responsibility that they’ve taken on is to share ideas across other teams,

all the people that like test or are interested in testing will belong to the testing guild, and then they’ll share great testing practices and testing strategies across teams and help each other to learn there’ll be or having coaches, and coaches going across multiple teams and helping teams also to interact better with each other as well.

That’s really where the coaches really take off and really have a huge impact is Helping teams just work together more effectively just for teams. There’s definitely ways to do that. But also understand, context counts.

So just because something works for your team might not work for my team or won’t work exactly for my team. And my team’s going to have to. Fiddle around with it. So for example, my software team might’ve figured out that continuous integration is a good thing. We, did everything we needed to do to set it up and got it going.

And then we share that with the data team and it would fail miserably because. Continuous database integration is significantly more complex than just continuous code integration, now it’s not that you can’t do it. It’s just, you’ve got to do it a different way because you’re dealing with persistent data, which the application development team wouldn’t be dealing with.

You need to remember that when the context changes, you need to expect that . Some things are going to work, some things will not going to work, or they’re going to work in a different way. You need to get into learning mode, basically, and experiment a bit and make it work for you.

And that takes courage and it takes skill. I don’t remember if I ever told you the database refactoring story, but years ago when we first started working on database refactoring as a technique, we basically, we didn’t know how to do it. But we knew it needed to be done.

And so we assumed that we could figure it out with no idea of how to do it. And that’s how we all stuck to it. And we experimented and finally figured it out. And it was a bunch of smart people that were working on it and we all had pretty good clues, but we still needed to work it through.

We could have easily given up and never figured it out. And this is the challenge, right? You got to stick to it. And toughen up, particularly when you see other teams doing it, it’s database refraction is a bit difficult because nobody was really doing it at the time.

, if you see another team and they figured out how to do continuous integration, you should probably assume you can figure it out too and that you’re not special and that you’re not in this weird situation where it’s not going to work out. It’s just that you’ve got to, your team’s got to figure it out and, make whatever changes that.

Shane: And go and ask that team, because most people will share, apart from the methodology people, they’re trying to see certification, so they have to follow all their content. We should. If we see a pattern, we should share it, because we’re using everybody else’s patterns anyway, 

and that, that reminds me, I was looking through the Big Book of Wow. A little book a while, and I got to page 28, which is the context diagram. And for people that haven’t read it, basically what it says is like a line and it has a left and a right, and there’s a bunch of lines, and that’s your context, 

you have something on the left and something on the right. And then depending where you fit, you get to choose. Patterns are applicable are more likely to work or they’re not. And it reminds me, I was coaching a team and we had to figure out their way of working, so we just came up with five lines of things that were important to them.

Do they want decentralized teams or centralized teams? How does the organization work? And on that slider, we put a dot where they felt they were, and then the patterns aligned with that. And I found that as a really great way of working with teams to start.

And if I’m scaling. If I know we’re going to scale, which we always do, actually, so we’re always going to scale, ? When we’re successful, we want more people to do it the way the successful team has. Going back to that slider and say the team made this decision to use this pattern, given this context, that is a great way of visualizing the reason they made that decision.

And are you in the same context? Or even go back to them because it’s been six months or years since they did it, and say, has the context changed? Has the pattern changed? So that visualization of why you’re choosing that pattern. For that part of your way of working I found a really valuable diagram and again, can’t remember if I saw it from you and then just used it or I saw it somewhere else.

Scott: It’s the scaling context factor. That’s something actually in Disponential Delivery back in the day at IBM. We were working on that in parallel with some work that Philippe Kruschen one of the original people behind RUP.

Was working on, he had something called the octopus model and had eight factors and we only had six. And so he was the octopus model and we basically released at the same time and we were talking to each other and sharing ideas. So yeah, those are the two sources there. And da, we continued to evolve it and tweak it a bit.

Whereas, thankfully, it moved on and started doing something else. I’m applying the same sort of thinking to data quality techniques. So what are the complexity factors or the, factors you should consider when identifying , data quality techniques for your situation? I’m still working on this.

And the problem is there’s so many data quality techniques that you got to rate them, but it’s just work. So anyway, so I’ve got a, I’ve got a growing database of this material. And what’s really interesting. is putting various life cycles into context. Lately I’ve been giving a a talk on machine learning and building up to a machine learning life cycle.

Here’s how it works in the real world, as opposed to what you maybe have been told. And then I started saying wait, so where are the data quality issues here? And so I started looking at the data flows basically, and for this data flow. These two factors of the five are critical the other three factors.

Yeah, they’re nice But I don’t really care about them. I really care about those two factors, but this flow over here I care about these two factors or these three factors and for obvious reasons, right? It’s always pretty straightforward and so Knowing that I care about these two factors, how do these 25 some odd techniques rate?

And I’ve basically got a Python code to generate a quadrant chart. So I said okay these… Three techniques are in the top quadrant. So I probably want to adopt one or more of these three techniques and these other techniques over here in the bottom quadrant are phenomenally bad ideas for this problem.

But then, Oh, but then I look at this line over here. And while these two factors that, Oh, look at this. So I, there’s four techniques that might solve this issue. And two of those techniques were in the bottom quarter on the last chart. So it tells you, You’re going to be, one quality technique’s not going to get the job done for you.

You’re going to have several because for this type of initiative, These are the issues and therefore you got to choose the right techniques for the job. Whereas if it was a data warehousing project sorry you got different quality issues there, or at least different quality priorities.

And as a result, you’ll have different data quality techniques if you want to be effective at it. It takes a while to build up to this, up to describing it, but it’s a really interesting way of looking at it because. You can rate the quality techniques on these five things.

And I really need a sixth factor, but unfortunately the sixth factor depends on the culture of your organization, which I can’t judge, but I can’t tell you how to judge it yourself and then throw it into the mix if you want to. And the idea is that. We can rate all these quality techniques, and then if you understand your life cycle that you’re following, then it’s a fairly straightforward analysis to, figure out here’s my priorities based on what on the way I’m working now.

Therefore, here’s how I’m going to choose these data quality techniques. And it’s a different set of factors than what we talked about in DA, because the factors in DA were of a higher level and you couldn’t get down into the weeds. But now because I’ve narrowed the focus down to data quality I can get down into the weeds with. Specific things to data quality that may or may not apply, actually a lot of them would apply for other issues, but these are the five factors for data quality.

Shane: That idea of allowing somebody to enter in their context and then have the machine recommend patterns that may fit their context that’s valuable, because as we talked about, we ended up with this pattern library that’s so broad, so massive, But there’s more we’re missing

so it’s not like we can never take anything away because every one of those patents has value. It’s figuring out how we can quickly say to a person, what’s your context? And then try these, cause we think these are the most likely. So just look at time just to close it out. You’ve been on a bit of journey, in your career from development through to coaching and writing up patents and building out the dad stuff.

And now. Back into education again,, what is your focus these days and how do people get hold of you if they want to talk to you about it?

Scott: So the easiest way to get a hold of me is just, drop by scottambler. com and it’s obvious once you hit the site, so I’m focused right now. I’m back to school for a master’s degree in AI. So that’s gonna keep me busy and it’s a part time degree, but that’s going to keep me busy until the end of 2024, and I’m doing some coaching, consulting part time as well mostly around data quality issues particularly in the machine learning space, .

Because everybody’s getting, it’s garbage in, garbage out, right? People are getting beat up on their data technical debt. I’m having a really good time doing that, and I do webinars for user groups and, chapters and all that sort of stuff, cause I enjoy doing that, 

Shane: I’ve always said sharing is caring and given the amount you’ve shared over the career that you’ve had I think that shows the amount you care in terms of sharing this knowledge and making it open and freely available to everybody. Again, big thank you for how much your content has helped me in my career but also just the fact that you share it.

So openly with everybody who wants to use it. On that note, , thanks for coming on the show, and I hope everybody has a simply magical day.

Scott: fantastic. Thanks for having me.