UX Research with Nick Fine

Alrighty so we start off by talking about what is UX research and you gave a great definition, which is the conversion of user needs into design elements. It’s user centric. It’s taking research into design. If there’s no user involved, you’re not doing UX research.

And then UX research is part of product. It should be part of the product management process, part of the product team, part of what we do when we build product.

Some organizations treat user research as the first step of a waterfall process. And that step is too long. And the comment you made is research is not a bus. It’s an iterative thing we need to do, but we need to do it. It’s not a big piece of work up front that we never go back and look at again. 

When I liken it to the data space, which is where I spend most of my time. In the data world, we’ve lost the art of data modeling. It’s too complex. It seems like a lot of work up front. It’s hard for us to determine or articulate our value. We asked to do it rather than say, it’s just the way we work. We don’t teach it in university anymore. And because of that, we’ve lost the art of data modeling. And I think UX research has fallen into all those same problem areas that we have with data modeling. It’s a lost art that had massive value, but not a lot of people are doing it anymore. Part of the reason that happens is commoditization.

So as something becomes popular, as it becomes valuable, it starts to get commoditized and we see people come from another domain and pretend they can do the job without actually learning the job. For you, it was market researchers have entered the market, watered it down. Coupled with this idea of the pandemic, removing one of the key tools that you have for UX research was observing a person in person. Because that’s where you can actually see what they’re doing without having to ask them what they’re doing. And yes, we’ve got digital, but it’s not as effective. 

We talked a little bit about research types, generative versus evaluative. Generative being discovering and then evaluative being understanding how it works. Which led me back to that original question I asked you about organizations that want to go out and research the problem space before they do anything, or organizations that go out and have a hypothesis on the problem and want to go and test both the problem and the solution space at the same time. And then actually we should be doing both. But we don’t do them linearly. We pick where we’re at, the context, what we’re trying to discover, what we’re trying to understand, what we’re trying to prove with data. And then we pick up one of those techniques or both, and we do them quickly and iteratively get.

Read along you will

Shane: Welcome to the No Nonsense Agile Podcast. I’m Shane Gibson.

Murray: And I’m Murray Robinson.

Nick: And I’m Nick Fine.

Murray: Hi, Nick. Thanks for coming on today.

Nick: Thanks very much for having me.

Murray: So we wanted to talk to you about user experience research today, but could we start by getting you to tell us a bit about yourself?

Nick: Sure. My name is Nick Fine, officially known as Dr. Nick Fine on LinkedIn. I am a 20 plus year psychologist in UX. I’ve been in practice since 2009. So I’ve been doing continuous UX for 15 years as a practitioner on project work every single day in both the public and the private sector agency, client side, enterprise, SME, the whole thing. I’ve been up, down and around the various blocks. 

Before that I worked in IT and I started off my career in networking and ISPs. But when the. com boom fell apart I went back into academia and then went into HCI. 

So I’ve been a journeyman for 15 years, going from town to town, getting into adventures and some fights and some amazing successes. What informs what you’ll hear today. Is just a lot of sleeves rolled up experience applied in different contexts.

Murray: Great. 

So the big question, what is UX?

Nick: So UX is the conversion of user needs into design elements through user research. The core point about UX and the differentiator from every other design discipline was research into design. UX is a user centric practice. So user centricity, user centered design, fundamentally defined what UX was and how it was performed. What today is a non user centric thing that calls itself UX. Which doesn’t have the same performance or results or activity. It’s just not UX.

Murray: I’ve worked with people who call themselves UX designers, but they hardly do any research. They do graphic design for web pages and software systems. And they’re great at it, but research doesn’t seem to be part of the package I

Nick: Yeah. That is a massive problem. When I talked about differentiation of UX design from any other forms of more traditional design or graphic design, print illustration, print design, whatever that is, Designers in a traditional agency will tell you that they do research but it’s a different form of research with a different intent. It’s there to inform color palette, emotion, those sorts of things. The research that a user research provides is absolutely about the user needs and the behaviors of the individual. And the design that happens is a behavioral design to those behavioral needs. That’s fundamentally different to making visual assets.

Murray: What’s the relationship between UX and product? 

Nick: Product is the headline. Things like UX research and design, content, service design are all pretty much children of product in that regard. And that’s how I see the world with the product manager leading that group. 

Murray: It’s very common these days for people not to have anybody in their team who is very experienced in UX research . If they have a UX designer on their team, they’re somebody who does designs mockups, and tend to do stuff in Figma. Sometimes I’ve seen people go out and hire a specialist UX research firm and they go off and do their own thing for a few months. And then come back and say, customers like this and they don’t like that. But what do you recommend that people do if they don’t have a UX research specialist in their team. . 

Nick: Ok, let me just start by saying there’s an awful lot of very poor UX research in the world. A lot of people’s experience of UX research isn’t of user research. It’s not of good, solid, efficient, valuable research. A lot of people finally get hold of a researcher, and then they get a really verbose, over heavy, opaque value report of stuff that doesn’t really move any needles or get into any consciousness or really help product manager shape their strategy or their roadmap or make any decisions. So there’s, probably a lot more bad research than there is good. And a lot of those teams have been cut and made redundant. So there’s a lot of very bad stuff going on out there.

Shane: Can you do UX if you don’t do research?

Nick: If the definition of UX is research into design, then no. The whole point about UX as a user centric practice is that it blends top down with bottom up. If you take away the bottom up, you’ve only got top down and that’s what you see in today’s world. Lots of unformed opinions or executions or low ROI stuff because somebody thought somebody else needed something and built it for them. And lo and behold, they didn’t need it. 

Shane: I see two types of organizations. There are UX research lead organizations who want to go out and discover the problem first and there are lean startup or MVP type organizations who want to shorten that cycle by guessing at the problem first and then going and testing the problem solution

Nick: This idea that you can put a hypothesis out into the world in some form or another, be it a wireframe or a sketch, or, just some cheap, dirty code to test it, learn from it, iterate it. That’s the center of user centricity and of agile .

This idea that we need to do this piece of research as a discrete lumpy thing is bad. I don’t support that. Of my 15 years in practice we’ve never had these big discovery periods of upfront stuff. It’s never existed. That’s all bleed over from market research and waterfall . That’s not UX. That’s not been my reality.

Shane: Which is the whole preface of Lean Startup. Build a solution for the problem, and go out and test both the problem and the solution space at the same time.

Nick: Shane, that to me sounds like reverse discovery. Which is I’ve built a thing and then go in the world and try to find the need for it.

Murray: There’s been quite a bit of research on feature usage on products and services by the Standish Chaos group who did it on internal applications and a company called Pendo who have done it on SAS applications. They both found that about 70 to 80 percent of product features are rarely used. And about 40 percent are never used. So really there’s only about 30 percent of the features you build, which are probably worth building. Does that gel with you?

Nick: Yeah. Because often the feature factory are creating features in a top down fashion that somebody thinks users need or want. Which is wrong. When you keep doing that, you end up with 80 percent of stuff that you don’t need and 20 percent that people might need because you’ve hit on actual needs. But if you were to map the user needs of the persona to the product the functionality should be echoing that you should be executing on the user needs or the pain points. 

Murray: Well, then you should be able to save two thirds of your time and money by just having some valid research on what people really need and want from your product.

Nick: Hundred percent, a hundred percent. 

People are also scared to delete stuff or take stuff away. If you can take stuff away you’re abstracting to a simpler, easier to maintain code base. Keeping things simple has a lot of value to it. People are incentivized by output, not outcome. So new features is output, not outcome. Whereas existing features that actually move the needle and help you make more money, that’s outcome. But that’s not sexy enough for anybody anymore, so they want new and shiny. And I think that’s a fundamentally big problem.

Murray: One of the agile principles is simplicity is essential.

Nick: Yeah, as a UX person, simplicity is one of my North Stars. But simplicity has been the casualty of the modern world. 

Murray: Is it your experience that what people are really doing is asking managers what they want, putting them into a big list and then building them rather than finding out what users want? 

Nick: Yeah or asking users what they want. And again, that doesn’t work out well either. It’s what users need, not what they want. And often users don’t know what they need. They’ll know what they want, but they aren’t the same thing. 

Murray: Can you do UX research at the same time as you’re developing the product with engineers?

Nick: Yeah, a hundred percent. And you should be. Research is not a bus that you have to get on because the next one isn’t coming for some time. Research happens every sprint as a BAU activity. If your PM or delivery lead is not facilitating that you need to elbow yourself some room. Talk about this. In a retro say. Let me do research on a regular basis. Or, I’m a big fan of the Maverick style. Go ahead, do it anyway. Don’t wait for permission. If you’re asking them to open their eyes, that’s a much harder sell. Just be the eyes and ears. Do it. If you do it, and you get great insight and there’s no harm done. You haven’t been illegal, immoral, or unethical. Then people will love what you’ve done. That’s called initiative. But if you’re waiting for permission and talking about, Oh, I can’t do it. And you’re moaning in retros, you’re going to be seen as this blocker, a pain in the neck. And that’s not a good look. 

Researchers should be a ninja helping the team out, getting you the eyes and the ears at the right time to make the right decision at speed. Instead, research has become this bloated, fat, horrible, large, slow, expensive thing that no one wants to do. Now I’ve been here before because in 2010, 2011, traffic managers, delivery leads, other people were cutting user research from projects, in waterfall, pre agile, because you could move the line to the left and save some money. That turned out really badly and we stopped doing that. And we’re right back here now and we’re building really shocking products with our eyes closed and wondering why people are getting fired. 

Shane: So do you think they’re not just taking the bull by the horn, going out and doing research, going and talking to customers? 

Nick: One of the problems is the loss of an explicit delivery role. And lots of inexperienced people don’t have the confidence to negotiate some room. My point being is it shouldn’t be a negotiation. And the reason why I made the point about delivery loss is because beforehand, everybody knew when to do their thing. And without that it’s all a negotiation. And some people like me are better negotiating their space to do their thing and other people are not. And that means it’s a casualty. It shouldn’t be down to quality of your negotiation.

Murray: Should you really have clear roles in a team or is it really important that you have the skills in the team? 

Nick: It really depends how important the thing is you’re doing or how much you really care about it. Usually in my mind, a role Implies doing the thing repeatedly, which implies repeated experience. And therefore you get good at it. Other people say they have the skills. If push came to shove, not a single one of them can do IA because they haven’t been practicing it. IA is a difficult skill and it takes some time to learn, but no one does it anymore and it’s dead. But yet everyone says they can do it. Or everyone says they can do research as a skill. Sure, I can do research when it comes to buying a new car, but can you find a cure for an illness? There’s different levels of research. 

Murray: The thing about thinking of a team as having a set of skills is that then you can be more flexible about what people do. So together the team has all the skills and sometimes that happens by people having two or three skills like a multi class character in a game.

Nick: Yeah, no, understandable. The idea of this multi talented individual is great. The only problem with this stuff is you can’t resource and staff with unicorns. And if you do, they’re really expensive, and when they leave, you’re stuffed. I don’t want to say no to people, but it has ramifications, and it’s hard to manage and staff and recruit when you’ve got to look for the skill, not the role.

Shane: Is the majority of the UX researchers who are skilled at it from a psychology background. 

Nick: No. The thing that blew my mind when I started in practice in 2009, was how few psychologists there were. And I’m looking around going, wait a minute, you guys are all doing experimental psychology, but completely amateur . So where are the pros or where’s anybody with the clue to help you guys do this properly? And there was no one. So in 2018. I put a video on YouTube called an introduction to UX psychology. It was a call to arms to anybody around the world saying, do you identify with me? Do you relate to the experience that I’m having? Cause this is surreal. I can’t believe this is happening.

It feels like I’ve come from the future. 

Shane: So if the majority of the UX researchers who are experts and coaches aren’t coming from a psychology background, where are they coming from? 

Nick: What I’m seeing are lots of market researchers. 

The pandemic was a massive impact on my practice on research because everybody’s working from home now, or most people are. The context of work has changed to home. That presents privacy and ethics limitations. I can’t watch you anymore in the office. User researchers were the great travelers of the world, because we went everywhere to do either some contextual inquiry, something ethnographic, interviews, whatever it was. Watching people do things is better than listening to them to do things every single time. But the methods that we’re forced to use post pandemic are self reported methods like surveys and interviews over zoom, which produces garbage, frankly, or limited results. And that’s a massive problem because I’m not building profitable and successful products on garbage and fairy tales, self reported constructed aspirational crap.

That’s what it’s all about. And market researchers, unfortunately, don’t have much of a concern for validity. That can kill me for that. I don’t care. When you do reach out for eyes and ears and you get somebody that’s too market research driven, it’s too fat, too macro, not useful, not connected. Can’t give that to a designer to build from.

Murray: I’ve heard people say when you interview people, they’re just trying to please you. Or they’re saying things that they would like to be true about themselves. It’s a waste of time interviewing people because the only valid research is with people using the real thing or close to the real thing and therefore you should only do MVP testing or mock up testing.

Nick: I totally agree with that. Self reporting is garbage. It’s always aspirational. Users by nature want to acquiesce to what they perceive the needs of the investigator or experimenter are. So you’re always trying to second guess, why are you asking me this? What’s the right answer? Those sorts of things. And that always happens. That doesn’t devalue the role of the interview in the toolbox, it just a major factor you have to handle. Interviewing is really good when you’ve got subject matter experts and you need to learn facts. 

So for example, I worked at the passport office, did a lot of interviewing of people because I’m learning about decision making factual stuff in doing that role. I couldn’t get it from observation or I’d have to ask somebody, why did you do that thing? 

Murray: Okay, just moving on a bit. Can you explain what information architecture is? 

Nick: It’s how information fits together. It’s the meaning of information, the semantics of it, and what it means to people. For example, there is a taxonomy of apples. That probably starts off with red or green. And the next level down is Granny Smiths or Braban. 

Pre 2016 2017, one of the very first activities that any UX consultant like myself would do would be to map the IA. We don’t do that anymore. We don’t even think about it. We have no concept of that. We don’t even do journey maps half the time . And all the journey map is one of the journeys through the information architecture. 

Murray: Journey mapping is great. Business process maps, journey maps, service design maps, user story mapping all seem to be around the same idea.

Nick: Yeah. Again, there’s too much fairy tales going on in our world and there’s too many people producing some really pretty, good looking stuff, which doesn’t mean anything. I love service designers, but service design maps, When they’re not created by a service designer can be works of pure fiction with no data in them and they lead product teams down the wrong path.

As you’re hearing, one of the big cornerstones of my practice is validity. I’m a scientist. Researchers should have a concern for validity and all these market researchers don’t have as much of a concern, in my observation, it’s much more checkbox exercise. It’s about completing the work rather than learning stuff.

Murray: All right. Let’s go into the different types of user experience research and when should you use them?

Nick: Okay, the simple headline split is between generative and evaluative. Generative is what most people consider discovery. I need to understand something about the world. Whereas evaluative is I’ve got a thing and I want to test it. I want to understand how it works. And the two go hand in hand. The big mistakes that happen is that we go in a bit. Waterfally when we go into discovery mode. There’s a lot of that going on. The way I see things are, these are in sprint modes to be called upon according to the needs of the roadmap item that you’ve been assigned. Not, this sprint is a discovery sprint, or this is an evaluative sprint. It’s, I have a discovery task, or I have a generative task, or an evaluative task which then plans accordingly.

Murray: What are the different methods in generative and when would you use them?

Nick: So generative is, I want to understand who’s using this thing? Who has a need to use this thing? What are their pain points in using this thing? What is the context of using this thing? You’re trying to describe or understand everything that relates to that product, service, and most importantly user. The generative stuff tends to be self reported. 

Pre pandemic, I could go down to big industrial building, stand around blending in, listening or see when they’re the activity, where the footfall is all the good stuff, which we can’t do anymore. So that’s the generative world.

The evaluative world is user testing. That means I’m putting something in front of an actual user or potential user. It could be a paper prototype, it could be some code, sometimes it’s Figma. And that means you can test the thing with the users and observe how they use it. That’s behavioral. 

Murray: Let’s say you’re down the track a little bit you’ve got a mock up and you want to Test it with users. How do you practically go about that? How do you get the users to come? What sort of users do you get? What do you say to them in the session? How do you analyze it afterwards?

Nick: So there are lots of different recruitment platforms out there. I would urge massive caution because they are all highly biased pools of paid participants. That means they are wildly biased and will say anything to get a couple of quid and you are not the only gig in town. So they are signed up to all these other services and will spend all day making 10 pounds every 20 minutes or half an hour every hour. It’s a legit income stream these days. My role today, is as much as a BS detector or lie detector than anything else because the signal to noise ratio has gone bonkers because of this stuff.

So, you need to recruit some people but try to make sure that they’re not wildly biased. You need to put this thing in front of them. There are services like usertesting. com who will allow you to do unmoderated user testing as opposed to moderated user testing and the difference there is where I set up a script in a system that says, step one, I want you to go to www. website. com. Step two, I want you to find a pair of red shoes that appeals to you and go up to the point of buying them. Don’t put in credit card type stuff. So I could be sitting here and be sent these tasks to do and do it on a recorded session. And that recorded session is then sent back to me, the investigator, and I get to go through hours and hours of footage, watching people buying red shoes.

It’s really super effective. It’s great. User testing. com. It’s good, but very expensive system and with any tool, it’s only as good as the person watching the feed writing the script and doing the analysis. It isn’t something that anybody can do, you can go quite wrong. The intelligence isn’t in the tool. The intelligence is in the user. 

If you want to test this stuff out for fun, do it with a colleague or a friend or a spouse . You want to have a script or some guide that will help you on that journey. Now, when you’re doing evaluative work, you’re usually testing a specific journey like onboarding, sign up, buying the red shoes, checkout, customer support, whatever it is. It isn’t just this wide go and use my thing. Let’s see how it happens. It’s a directed inquiry. 

So we asked them to do a thing, go do the things you watched them doing it. Ideally, you might want to record this. If it’s done over zoom, or meet, it’s easy to record. And then you watch people follow the script. When I say script, it’s not a verbal script. It’s a action script. Make sure they cover these items of the journey. So for example could you try using expedited shipping or log into my account to increase the monthly limit. 

Then you go through and analyze. What you’re looking for are user needs and pain points. That’s what everything that I and most good user researchers do. Everything is boiled down to a user need or a pain point, because that’s all I care about. 

The big problem you’re seeing in market research land is people are going fishing and they’re catching everything in these big discovery nets. They’re catching whales and sharks and dolphins and tuna and everything. When what they were asked to do is to go catch tuna. So they’ve gone out and they’ve caught the tuna, but they’ve also caught this other stuff and they’re reporting on it. But that’s ridiculous. We didn’t ask you to go do that. That’s not what your mission was. That wasn’t the research objective. So the fact that you’ve come back with two massive whales and you’re all excited about it, so what? It’s not the tuna that we ordered.

Murray: What user recruitment platform do you recommend? 

Nick: Ah, testing Time is the people I’m going to say. Cause I’ve worked with them in the past. They are a reasonably decent participant pool. I haven’t worked with them for the past few years, but give them a go. Just be aware, handling paid users is dangerous, but it is the way of the world. So you have to unpick the signal from the noise.

Murray: How much do I have to pay people to do research with you?

Nick: It depends. I was working with C level folks a couple of weeks ago. I was paying them a hundred quid for a 30 minute session. It seemed to work and I got some really great people. When I’ve worked in the past with ultra high net worth individuals, they wouldn’t get out of bed for a hundred quid , it needs to be a lot more than that. I’ve been working on other projects where 50 quid does the job. When we were in the UX lab before the pandemic you’d pay people 20 to 40 quid plus travel expenses. It really depends upon the sort of persona or segment you’re trying to hire and the nature of them, if they’re ultra senior or ultra wealthy, it’s going to cost you a ton more cause they’re time poor and they don’t really care for what you’re doing.

Otherwise if it’s lower grade, then it’s much easier to recruit. Most people’s work tends to be that lower grade which means you get everybody piling in and will say anything. Case in point, I just recently wanted to recruit AI users. How do you screen an AI user effectively? That’s a tough gig. I did well, but that requires some experience. Someone without the experience would get taken by the crowd big time.

Murray: Going on to the actual session where you’re interviewing people. What approaches are there for the interview part?

Nick: Yeah. Okay. So the style that I like to do is the blank slate style. I’m a behaviorist. So I’m looking for behavior. The more I lead the witness or talk or tell them or bias them, the more I’m taking away the natural behavior, the natural reactions that are so indicative to me. So I would much rather throw them in the deep end. I don’t want to give them help. I want to see what happens because when you’re using software for the first time no one is supporting you. There’s no one there. You’re having a solo one on one experience with the application that you’ve built or product, or service. And that’s what I want to try to mimic in the lab or in a user testing session, because I want to see, I don’t want to be the bias. I’m not going to be there. 

However, there’s a second style where it’s, how do you feel about that and all of that? This is where things get spicy and a bit more interesting. Figma is an amazing design tool with fantastic UX built into it. A few little niggles, but it’s industry leading and brilliant. It’s crappy at prototyping and it doesn’t really compare to something like Axure.

What that means is it produces limited prototypes that don’t really elicit or display actual interactive behavior, because you can’t actually interact with it because it’s limited. That means you end up using it much more like what market researchers would call a stimulus. And that’s fine. That’s how I use Figma these days as a more of a generative piece rather than evaluative piece, you can’t do evaluative with it. You can’t, it doesn’t have the interactivity. When you’re asking people, what would you do? That’s not behavior. That’s constructed fairytales. So I can’t get that with Figma of the current version. So I use it as a stimulus to say, here’s a tangible thing, let’s talk about it. That’s got value to it. You still gotta pick the meat from the bones and the signal from the noise. It gets messy, but it’s a hell of a lot better than trying to fake out Figma as an evaluative journey. It ends up being a stimulus, whether you like it or not, call it what it is and treat it appropriately. That’s how I look at that.

Shane: So there’s a bunch of tools that record screen interaction for the users automatically. Those tools are now using AI to sieve through and find cohort of behavior. So these users spend a lot of time over here and they’re different to the group of people that look like this. What’s your view on that? 

Nick: I’ve sat in multiple of my clients chairs with the web analyst looking at these sorts of tools. Here’s a good example. You’re looking at scroll depth. And it says, nobody scrolls deeper than 80%. And you look at the actual map and at 190 percent there’s dots everywhere. So it’s internally contradictory. It’s got questionable validity. Even Google, works on extrapolated data. That means it’s guessing.

You’re going to base your business Intel on that. There are people who are just accepting these tools as gospel. They’re not even questioning it. And then they’re building stuff and planning stuff on it. I think that’s phenomenally dangerous.

Murray: I want to ask more about validity and bias from an interview. Can the interviewer lead the subject in the direction they want to go to produce the results that they want?

Nick: Yeah, totally. You really shouldn’t do that stuff. We want natural behavior. It’s all about the user. But there are time constraints. Sometimes, people like me who are neurodiverse go off all over the place and you’ve gotta keep them honest, which means cutting them off, putting them back on the rails. As a researcher, I’ve got a research objective in my head. There are things I need to learn, so I’m gonna have to muscle you around a little bit. And usually with light gloves, so you don’t even know when I’m doing it because I’m trying to get what I need, not what you want to tell me.

And there’s a massive difference there . I worked with a guy who I lovingly called derp because he was such a moron. He was like a secretary. He just sat there in front of people and recorded everything that they said. There was no inquiry. There was no probing. There was no checking, no nothing. That’s not research. You’re being a recorder.

Murray: How important is empathy and an emotional connection with the subject?

Nick: It’s very important. Your empathy helps you be sensitive, helps you understand the pain points and the needs of people. If you don’t feel something for them and you don’t want to make their life better, then you’re in the wrong job. 

But I don’t believe that it’s ultra critical. If you’re on the spectrum and empathy is a problem for you, or you don’t experience it in the way that other people do, you can still be a bloody great researcher, 

Murray: What about the jobs to be done approach, which seems like a much more rational, task focused approach?

Nick: Jobs to be done is for me a problem. Because jobs to be done, I think over engineers the concept of user need. I think it’s an absolutely brilliant way that Christensen characterized the use of user needs. And I love the milkshake video. That people employ a thing to do a job for them. The way that I look at it is I have a need and as a user, I need to do a thing so that I can do a thing. So as a driver, I need a milkshake so that keeps me company, keeps me alert and keeps my stomach full on a long drive. Brilliant. 

However, and there’s a lot of parallels with agile and products. Jobs to be done became super popular a few years ago and become super popularized and every management consultant and mentor and coach just got hold of it and started training it because it sounds good. But they overcomplicated it to get social needs and all the various levels of needs is way too far. And because it’s so hard to get hold of this stuff and we haven’t got competent researchers en masse, people make this stuff up. And the jobs to be done become fairy tales. And you can’t build products on fairy tales where you can, but it’s not going to be a good product.

Murray: How many people would you have to interview to get some sort of valid result?

Nick: There was a study that Norman and Landauer, or maybe it was Nielsen and Landauer, put out that said. In 19 66, that you can get 80 percent of the problems from eight to 12 users. And in fact, you’ll probably get it after three or four, if it’s a really big problem. The problem is that people misinterpreted that and they think I’m only gonna do five to eight people.

And if you’re testing a product with five to eight people and then releasing, you’re in big trouble, massive, crazy trouble. You should be doing this as a regular cadence. It should be part of what you do so that five to eight people you’re getting every iteration, every sprint. That’s the best way to do it, the safest way to do it, but just don’t make it big and heavy, make it small and light and regular.

And that way it’s less painful and much easier to do low cost. And the most important part for the product managers and for the agilists out there, it keeps you honest. It’s course correction. If you’ve got eyes and ears every sprint, every week. You’re all honest. You’re all kept honest by the data, by evidence instead of being in sprint planning, being in a retro, having long extended discussions about what should we do, forget that stuff, get the evidence and let it lead you. Create hypotheses, investigate those hypotheses, do it small and frequently. That’s the way forward. That’s the beautiful thing about agile that got broken. 

Murray: What is the best way to share your results so that you can have an impact? 

Nick: Murray the two things that I do these days are. One page persona, and showreel. When I say showreel, a short video of two to three minutes, maximum. What moves the needle in my world, in my experience are those two assets, nothing else. 

For me, a persona is a data container of insight. Always. So, If it’s a user need or a pain point , or it’s contextual, it goes in there. Now, this is beautiful. Cause you’ve got a one page piece of paper that should have only gold in it. Only juice. No big insight reports. None of this tomes of garbage that we used to produce at agencies to keep the client happy because it was a perception of value. None of that stuff. That’s the leanest way of transferring the knowledge or the insight that you’ve learned.

Now, remember, we’re just putting down the user needs about the stuff I was tasked to learn, not, Oh, by the way, I learned about whales and sharks and eagles. If you need to do whales and sharks and they get their own persona. 

Murray: What about having the team observe the interview in real time with you? Or some of the team, like the product owner or manager or whatever.

Nick: So that’s a really dangerous thing to happen. Because what happens is they don’t shut up and they jump in. Because they’re the product manager and they feel like they’re entitled to do so. It happens a lot.

Second of all, if you’re a participant, you do not want to be sitting in front of a committee of people looking at you. It affects your behavior. It affects the way you feel about the session. It changes your performance effectively. So I’ve been in rooms where I’ve watched a delivery lead, a product manager, an engineer, a marketing person, all looking at this one bloody user.

What is more effective is make the raw unedited recordings available to your product team. That’s what I do. That means you get the full session. You can muck around, you can see the whole thing, but you can’t bias it in any form whatsoever. It’s much safer that way. 

Murray: Another way I’ve seen it done was as a customer journey map or a user story map with. Pains and gains for each step from your research. And maybe a quote. To illustrate it. 

Nick: As long as it’s based on data. If you know that people are happy or sad at that step, because you’ve got data on it, then great. But if you think that you’ve seen one or two people struggle at that stage, and that’s now your insight, that’s a major risk. And the question is, are you or the product team willing to roll those dice? That’s not for me to say. Some people are in situations with high pressure and time and market, got to get to market and all of that stuff. 

Murray: We’ve focused on watching people use something or a version of something, maybe Figma, maybe something else. What other research methods that you would use? 

Nick: The best value is user testing I try to harp on about this as much as I can on LinkedIn, because I believe in it so strongly and I actually posted on it last night. Our world is messy right now. And people are looking to make things better or to move the needle in some way, because it’s a really tough world. And the answer to that right now is user testing. It’s just cut straight to the user test. Take whatever your product or service is test the golden journey, the main thing that makes you money, the main thing that you need people to do well. Watch them doing it in numbers, five to eight to 10 people. Where there are problems, they’re going to come out pretty quickly and you’re going to see them. Now, if you’ve never user tested these journeys before, I know you’re going to have big problems. And that’s the opportunity to move the needle. Please look at BAU functionality as important and valuable, as new functionality. It will help you guys out massively.

The world today is all about new and shiny, and it’s been causing us all kinds of shenanigans because we’ve been looking at extending our house and putting new fancy things on without looking at the foundations. We have to do that immediately. And that means testing the core things that make you money and not interviews or surveys, watching them do it.

Now, if you’ve got no research experience whatsoever. And you want to just do a quick test to find out what’s the health of my product or service. Get five people, make sure they’re not internal people or friends or family. Don’t say anything, stick it in front of them, watch them using it for five minutes. Just see what happens. You’re not a pro researcher. You’re going to make all kinds of mistakes. That’s okay. But you’re going to see some pretty natural behavior quite upfront. And when they go don’t know what this thing is. You’re going to have a really big wake up call. 

Murray: Can you tell us stories of where you’ve done research and found out something that made a really big difference to a product or a service 

Nick: Yeah, I can tell you those stories all day and all night. The one that jumps to mind was on a multi million pound project in recent years, and I can’t name names. They built a system that required the other side to nominate an administrator. So if you buy our service, you’ve got to tell us who the admin is so that we can then set them up for security purposes.

Everybody from the engineering product teams were like, yep, That’s the solution to this, it’s fine. We’ll just make them give us an admin. So I’m out there testing this thing for the first time and everybody comes back. I don’t want to be the admin. You’re not paying me to be the admin. I don’t have time in the day to be the admin. How many emails is this thing going to generate as the admin? So there’s a massive assumption made that we built to. We built a whole big system. They’d already built this system before they brought me on as a contractor. So we were like a year in and they’re like, Oh yeah, now let’s test it. And the first testing, this is what came out. No one wanted to be admin. This is an admin based system. 

Murray: So what happened 

Nick: It hit the wall in a big way. I left the project and I’ve heard from people since that it has not gone well.

Murray: Have you got any positive stories? 

Nick: Yeah. , I’m in the lab and it’s 2012. I’m in there with Amy, my colleague, and we did tons and tons of work back then. We’re doing work with senior folk looking at retirement homes and all that stuff. We’re in the lab, the lady comes in, lovely blue rinse old lady with jam jar glasses on, and she sits up and she’s trying to see the screen. And I said to her I’ll pull the screen closer to you, but do you know what, we know that people like you with limited vision are going to be using this website because it’s for people like you we’ve built something on page to help you make the text bigger, can you please go and find it? She went to the magnifying glass in the search bar. The point being is there’s a lot of serendipitous or lots of really interesting things that you learn from user research that helps make things better for people.

If people have got limited eyesight and they’re looking for a magnifying glass, and if you’re a certain age, that’s probably the right interaction design. These things happen. This idea of one and done is mental. It’s never one and done. But yet somehow this idea that you do this one and done and get it right first time is in the professional psyche.

And it needs to go away. Things take time and practice and failing and learning but people are like, yeah, I’m done. Check box. Build it. Ship it.

Murray: I think we better wrap it up . Shane do you want to kick us off with a summary? What do you got?

Shane: Alrighty so we start off by talking about what is UX research and you gave a great definition, which is the conversion of user needs into design elements. It’s user centric. It’s taking research into design. If there’s no user involved, you’re not doing UX research.

And then UX research is part of product. It should be part of the product management process, part of the product team, part of what we do when we build product.

Some organizations treat user research as the first step of a waterfall process. And that step is too long. And the comment you made is research is not a bus. It’s an iterative thing we need to do, but we need to do it. It’s not a big piece of work up front that we never go back and look at again. 

When I liken it to the data space, which is where I spend most of my time. In the data world, we’ve lost the art of data modeling. It’s too complex. It seems like a lot of work up front. It’s hard for us to determine or articulate our value. We asked to do it rather than say, it’s just the way we work. We don’t teach it in university anymore. And because of that, we’ve lost the art of data modeling. And I think UX research has fallen into all those same problem areas that we have with data modeling. It’s a lost art that had massive value, but not a lot of people are doing it anymore. Part of the reason that happens is commoditization.

So as something becomes popular, as it becomes valuable, it starts to get commoditized and we see people come from another domain and pretend they can do the job without actually learning the job. For you, it was market researchers have entered the market, watered it down. Coupled with this idea of the pandemic, removing one of the key tools that you have for UX research was observing a person in person. Because that’s where you can actually see what they’re doing without having to ask them what they’re doing. And yes, we’ve got digital, but it’s not as effective. 

We talked a little bit about research types, generative versus evaluative. Generative being discovering and then evaluative being understanding how it works. Which led me back to that original question I asked you about organizations that want to go out and research the problem space before they do anything, or organizations that go out and have a hypothesis on the problem and want to go and test both the problem and the solution space at the same time. And then actually we should be doing both. But we don’t do them linearly. We pick where we’re at, the context, what we’re trying to discover, what we’re trying to understand, what we’re trying to prove with data. And then we pick up one of those techniques or both, and we do them quickly and iteratively get. So that was me, Murray, what do you got?

Murray: Yeah this is good, I think it’ll be useful for a lot of our listeners who aren’t UX researchers to be able to have some confidence that they can do this stuff themselves. Now, I know there’s probably a lot of UX researchers would be very unhappy that we’re saying that the team could do this themselves. But if you don’t have a UX researcher, then you should still be doing UX research the best way you can. 

Shane: But murray any other role in digital or it? How would you guys feel about other people having a go at your job?

Murray: It’s happened plenty of times and I don’t mind. I might argue that an organization hire somebody who’s a specialist, but if they’re not going to, I just work with what I’ve got. And so I’ve quite often coached, mentored and trained people to go some way to what I’m doing. It’s nowhere near as much, obviously. But it’s still quite useful and helpful for the organization, and maybe they will end up hiring somebody once they’ve seen some value in it.

Nick: Yeah. But the nature of insight and the good that it could do or the harm that it could do is very much unlike any other factor in our world. If you get research wrong and insight is wrong, you will have a massive impact on ROI, on brand reputation, on market share. Big things. We usually have professionals to do this. Research insight is important. The question is how much do you value the insight? Is having DIY eyes and ears okay for your big brand in the enterprise or wherever you are? Are you acting like a startup when you’re actually an enterprise person who should have a dedicated professional who is qualified and experienced to get you good golden insight? Or are you going to have some product managers doing DIY stuff and just hoping that this stuff turns out nicely? Up to you.

Murray: If I was in charge of the budget, I would hire a UX researcher to work in the product team with the product manager and UI designer and with the engineers. 

But that’s not happening very much these days. 

Shane: I really do understand that. I really do, but I would like to see the product managers asking how do you know this stuff? Is it because you’ve had an opinion, you had an idea? Because if I’m giving you money I want to have some indication that it’s being spent wisely and safely.

Murray: Yeah. But do you think that some research is better than no research?

Nick: That is the age old question. You can do some really bad research and go off in the wrong direction. Or if you’re lucky hit gold. It’s really a risky thing. Are you prepared to take that risk? And some of these people we’re talking about and some of the companies that you guys will work for are really big brands acting really irresponsibly.

Murray: Yeah. The really big ones to tend to have user experience groups, but they’re full of UI designers these days. So I don’t know. Look, you’ve taken us through the process of what you could expect. If you were a product manager. Here’s what you should expect from a UX researcher on your team. And if you don’t have one, maybe you can do some of it yourself with your team.

Nick: Yeah, but know that in the world today, there are a lot of very competent senior user researchers available for hire, but people do not want to hire them either because they’re too old and there’s a lot of ageism. So the talent is there. The knowledge is there. We’re not connecting it to the product teams. Most people are being excluded. Not everybody has my level of expertise, but there’s loads of people out there with my level of expertise who are available today.

Murray: I don’t really know why people aren’t hiring more UX researchers. I think maybe one reason is management thinks that UX and UI are the same thing. So they try and hire one person to do both. And that person rarely has research skills. 

Nick: I think we need to understand the true value of research and insight rather than this whole top down feature factory, which is the trajectory we’ve been on.

Murray: Look, I think the reality is, if you have 50 or 60 people building a product or a service, then you’re probably only going to be able to hire one UX researcher to sit up the top in your program team. And that person is going to have to coach and mentor and support other people in the teams who want to help and get involved. So a UX researcher not only has to be a professional, but also an educator and a coach in this world, I think, realistically.

Nick: Completely agree. Yeah. 

But, only the seniors are comfortable and competent enough to do that, and again they’re all excluded. So that isn’t happening, Murray. 

Murray: Yeah, that’s a shame. I think we’re all getting to that age now 

Nick: It sucks. We all know the right way, but we’re not allowed to do it. That’s the big frustration in the world.

Murray: Yeah, it is. why don’t we tell people where they can connect with you and Engage with you. 

Nick: Yeah, sure. LinkedIn is the number one place where I’m the most noisy. That’s where I dump all of my thinking. Connect with me, follow me on LinkedIn. The other thing I would encourage you to do is to look at that introduction to UX psychology video on YouTube.

We will come back around to that. And if you want to be a more visionary PM have a look at that video. I was way too far ahead of the curve, but once we’ve bottomed out this current bear phase of the market, when we start picking up again, we will be picking up at the intersection of behavior, design and research, which is what UX psychology is. So most product folk, that’s the best thing I think you can do is start to look ahead at that prize. As we’ve discussed, behavioral is great. We just need to work out ways of integrating into our ways of working to produce better products and services.

Murray: And you’re writing a book called Rapid Research?

Nick: Yes, I am. Rapid Research is going to be a book which is about stopping the overworking of research. Because there is too much boiling of the ocean going on. Teams are losing their jobs and we’re not delivering value. I’m going to put down my style of user research, which I’m calling rapid research. Yet retaining a good, strong eye on validity. So how do you get the good stuff? How do you get the gold quickly, cheaply, easily.

Murray: Great. Thank you very much for coming on nick.

Nick: Thank you so much for having me guys. Amazing questions, Murray. Really great. Shane, thank you so much. It’s been brilliant. So thanks again.