Is Your Company Reading Data the Wrong Way?

CURT NICKISCH: Welcome to the HBR IdeaCast from Harvard Business Review. I’m Curt Nickisch.

You’re a business owner and you’re interested in reaching out to new customers. You know that data is important. I mean, that’s clear, right? So you put out a survey into the field asking what kinds of products your ideal customers are looking for. You get that data back and you have a clear decision made for you as to which direction to go. You develop and sell that new product with a big marketing push behind it and it flops. But how can the data be wrong? It was so obvious. Today’s guests believe in data, of course, but they see major ways in which over reliance or under reliance on studies and statistics steer organizations wrong.

Whether it’s internal or external data, they found that leaders often go to one of two extremes, believing that the data at hand is infallible or dismissing it outright. They’ve developed a framework for a better way to discuss and process data in making business decisions, to interrogate the data at hand.

Michael Luca is a professor at Johns Hopkins Carey Business School, and Amy Edmondson is a professor at Harvard Business School. They wrote the HBR article “Where Data-Driven Decision-Making Can Go Wrong.” Welcome. Thanks so much to both of you.

AMY EDMONDSON: Thanks for having us.

MIKE LUCA: Thank you.

CURT NICKISCH: So are business leaders relying too heavily on data to make decisions?

AMY EDMONDSON: I don’t think that’s quite the problem. One of the things that really motivated Michael and me to get together is that I study leadership and leadership conversations particularly around really difficult, important decisions. And Michael is a data science expert. And our mutual observation is that when leadership teams and leaders are using data, or teams at any level are using data, they’re often not using it well. And so we’ve identified predictable or frequent errors, and our idea was to help people anticipate those and thereby do better.

CURT NICKISCH: Is it more of a data science understanding problem here or more of having the right culture to discuss the data correctly?

AMY EDMONDSON: Well, that’s just it. We think it’s both. But I’ll just say, in a way, my side of the problem is we need to open up the conversation so that it’s more honest, more transparent. We are in fact better able to use the data we have. But that’s not enough. And that’s a lot, but just getting that done will not ensure high quality data-driven decision making.

CURT NICKISCH: Mike, data has kind of been all the rage, right? For at least the last decade. I feel like it was 10 years ago or so that Harvard Business Review published this article saying that data scientist was the sexy new job of the 21st century. A lot of places make a priority of data to have something concrete and scientific. If they’re getting better at collecting and analyzing data, where’s the decision-making problem here?

MIKE LUCA: We’re certainly surrounded by data. There’s growing data collection at a wide variety of companies. There’s also growing research that people are able to tap into, to try to get a better sense of what the broader literature says about questions that managers are grappling with. But at the same time, it’s not really about just having data. It’s about understanding both the strengths of the data that you have and the limitations and being able to effectively translate that into managerial decisions.

There are a couple of challenges that we discussed in the article, but they all come down to this idea of once you see an analysis, and the analysis could be coming from within your company or from something that you’ve read in the news or from a research paper, how do you take that and understand how that maps to the problem that you have at hand? And that’s the decision challenge. And this is where effective conversations around data and having a framework for what questions to be asking yourself and what questions to be discussing with your team come into play.

CURT NICKISCH: In your interviews with practitioners, you identified that there was kind of two big reactions to this data that’s been collected, internal or external, as you just said. Where did those reactions come from? Why are we seeing that?

AMY EDMONDSON: As you said, Curt, data is the rage. Everybody knows today we need to be using data well, maybe we should probably pay attention to the literature and be managing according to the knowledge that exists out there.

CURT NICKISCH: And we have more than ever.

AMY EDMONDSON: And we have more than ever, right? So you can really understand the, “Okay, great. You’re telling me there’s the answer. Everybody should get a pay raise and that’ll make us more profitable. Okay, I’m just going to do it.” Or “Yeah, that’s nice literature out there, but really we’re different.”

I think we see both modes and they’re easy to understand. Both are wrong, but both need to be more thoughtful and probing in what applies, what doesn’t apply, what does this really mean for us? And we believe there are good answers to those questions, but they won’t pop out without some thoughtful conversations.

MIKE LUCA: Analytics or any empirical analysis is rarely going to be definitive. I think the conversations need to come around, what are the outcomes that we’re tracking? How does it map to the things that we care about? What is the strategy they’re using to know if an effect that they’re saying is causal actually is? And I think those conversations often don’t happen, and there’s a number of reasons that they don’t happen in organizations.

CURT NICKISCH: So you’re advocating for this middle path here where you really interrogate the data, understand it, understand its limitations, and how much it does apply to you, how much it can be generalized. Which sounds like work, but you’ve laid out a framework to do that. Let’s start with where the data comes from, internal or external, why is that a key thing to understand?

MIKE LUCA: When we think about external data, there’s exciting opportunities to take a look at what the literature is saying on a topic. So for example, suppose that you are managing a warehouse and trying to understand the likely effect of increasing pay for warehouse employees. You don’t have to just guess what the effect is going to be. You could take a look and see other experiments or other causal analyses to try to get a sense of what people have learned in other contexts, and then you as a decision maker could think about how does that port over to your setting.

Now in thinking about how to port over to your setting, there are a couple of big buckets of challenges that you’ll want to think about. You want to think about the internal validity of the analysis that you’re looking at. So meaning was the analysis correct in the context in which it’s being studied? So is the causal claim of wages on say, productivity, is that well identified? Are there outcomes that are relevant there? And then you want to think about the external validity or the generalizability from that setting to the setting that you are interested in and think about how closely those map together.

So I think it’s both an opportunity to look more broadly than what the literature is saying elsewhere and to bring it over to your setting, but also a challenge in thinking about what’s being measured and how to port it over.

Now, for larger companies especially, there’s been a growth of internal data. So you could think about Google or Amazon or other large tech companies that are tracking exorbitant amounts of data and often running experiments and causal analyses. Those come with their own challenges thinking about what is the metric we care about?

So it’s slightly different challenges, but related. But then zooming out, what you want to think about is combining what internal and external data do we have and how do we put it all together to come to the best decision that we can

AMY EDMONDSON: To get a fuller picture, really. In a way, what we’re saying, which is pretty simple, but I think really profound, is that you can’t just assume, if someone tells you, “Here’s a result,” you can’t just take it at face value. You have to interrogate it. You have to ask questions about causality. Was it an experiment or not? You have to ask questions about what was actually measured and what’s the context like and how is it different from my context and all the rest? And these are things that scientists would naturally do and managers also can do and get better decisions as a result.

CURT NICKISCH: It’s a lot of basic statistic skills, right?

AMY EDMONDSON: Yes.

CURT NICKISCH: That everybody has. It sounds like you kind of want that capability across the team or across the decision makers here, and not to have this sort of only housed in a data analytics team in your group, for instance.

AMY EDMONDSON: Yes, and – it’s not that everybody needs to be a data scientist, it’s that data scientists and operating managers need to talk to each other in an informed and thoughtful way. So the managers need to be able to learn and benefit from what the data scientists understand how to do, and the data scientists need to think in a way that is really about supporting the company’s operations and the company’s managers.

MIKE LUCA: Maybe just one quick example: this famous eBay experiment that looks at the impact of advertising on Google. And what they found is largely the ads that they had been running were not effective at generating new business coming in to eBay.

CURT NICKISCH: And just to spell out this eBay experiment, they found that they had been advertising in markets and seeing more sales there, and they thought the advertising was working, but basically they were basically just advertising to people who were going to be buying more from them anyway, so the effect of all that advertising spending was pretty muted.

MIKE LUCA: Yeah, that’s exactly right. So they had been running billions of dollars of ads per year on search engine ads. And so they had actually brought in consultants to look at this and try to analyze what the impact was. And initially they had thought that there was a positive effect because of the correlation. But then by thinking more carefully about the fact that ads are highly targeted, that led them to run an experiment to get at the causal effective ads. And that’s when they realized that many of the ads they were running were largely ineffective.

CURT NICKISCH: And so was this a correlation causation problem essentially at its core?

MIKE LUCA: So for eBay, there was a correlation versus causation problem. Then you could think about generalizing that to other settings, other types of ads on eBay, other companies that want to use this result. In fact, even within that one experiment, when you dive a little bit deeper, they found certain types of ads were slightly more effective than others. So you could find corners of the world where you think there’s more likely to be an effective advertising and change your advertising strategy.

So it’s correlation, causation, and then trying to learn more about mechanisms or where ads might work so that you could update your strategy. Then as external companies saying, “here’s this new evidence that’s out there, how do I take this and adjust either my advertising strategy or my approach to measuring the impact of advertising?”

CURT NICKISCH: Tell me more about the disconnect between what is measured and what matters. We all know that you get what you measure. We’ve all heard that. Where do managers often go wrong here?

MIKE LUCA: Such a challenging problem. And actually earlier we were discussing the fact that many things are measured now, but many more things are not measured. So it’s actually really hard to think about the relationship between one empirical result and the actual outcomes that a company might care about at the tail end.

So for example, so imagine you wanted to run an experiment on a platform and change the design. You change the design and you see more people come. That’s one piece of the puzzle. But you really want to see what’s the long run effect of that? How many of the customers are going to stick with you over time? How happy are they with the products or the engagement on the platform? Are there going to be other unintended consequences?

And those are all really hard things to measure. We’re left in a world where often analyses are focused on a combination of important things, but also things that are relatively easy to measure, which could lead to omitted outcomes either because the challenge of measurement or because somebody didn’t think to measure it. And that could create pretty important disconnects between the things that are measured in an experiment or an analysis and the outcome of interest to a manager or an executive.

CURT NICKISCH: Amy, when you hear these problems like disconnects – could also call that miscommunication.

AMY EDMONDSON: Absolutely.

CURT NICKISCH: From an organizational culture perspective, how are you hearing this?

AMY EDMONDSON: So I hear it as I think there’s a general need to go slow to go fast. And there’s a strong desire to go fast just in everything, data, it’s a modern world, things are moving fast. We want to get the data and then make the decision. And we write about the fact that it’s this issue we’re talking about right now that making sure that the outcome we’re studying, the outcome we’re getting data on is in fact a good proxy for the goal that we have. And just that getting that right, then you can go fast, go faster. But really pausing to unpack assumptions that we might be making: what else might this design change encourage or discourage? What might we be missing?

Asking those kinds of good questions in a room full of thoughtful people, well, more often than not, allow you to surface underlying assumptions or things that were missing. And when a culture allows, when an organization’s culture or climate allows that kind of thoughtful wrestling with very ambiguous, challenging, uncertain content, you’ll be better off. You’ll design better experiments, you’ll draw better inferences from the data or studies that you do have access to.

CURT NICKISCH: We’ve talked about the disconnect between what’s measured and what matters, conflating correlation and causation. Let’s talk about some of the other common pitfalls that you came across in your research. One is just misjudging the potential magnitude of effects. What does that mean? What did you see?

AMY EDMONDSON: Well, we talk about our general lack of appreciation of the importance of sample size. Certainly, any statistician knows this well, but intuitively we make these errors where we might overweight an effect that we see in a very small sample and realize that that might not be representative to a much larger sample. So how precise can we be about the effect that we’re seeing is very much dependent on the size of the sample.

CURT NICKISCH: You suggest a question to ask there, what’s the average effect of the change to get a better sense of what the real effect is…

MIKE LUCA: I think for managers, it’s thinking about both what the average effect that was estimated and also what the confidence interval is to get a sense of where the true effect may lie.

And thinking about confidence intervals is important, both before and after you conduct an analysis. Before you conduct an analysis, anticipating the uncertainty and effects is going to tell you how large of a sample you might need, if you’re going to say run an experiment.

After an analysis, it could tell you a little bit about what the range of true effects may be. So a recent paper looked at advertising experiments for variety of companies and found that many of the experiments that were being run didn’t have the statistical power to determine whether it had positive or negative ROI.

AMY EDMONDSON: So they’ll hear, “Okay, sales were up 5%. Oh, great, let’s do it. Let’s roll it out.” But in fact, that up 5% was well within what’s called the margin of error, and may in fact even be negative. It’s possible that advertising campaign reduced interest in buying. We just really don’t know based on the sample size.

CURT NICKISCH: Overweighting a specific result is also a common trap. Can you explain that?

AMY EDMONDSON: Yeah. It’s a confirmation bias or a desirability effect or we see something, or sometimes if a result is just very salient or it kind of makes sense, it’s easy to just say, “Okay, this is true,” without pressure testing it, asking what other analysis are there? What other data might we need to have more confidence in this result? So it’s kind of a variation on the theme of the magnitude of the effect.

CURT NICKISCH: One common pitfall is also misjudging generalizability. How problematic is this or why is this problematic?

MIKE LUCA: So we talk about that example in the article where there’s an SVP of engineering that was talking about why he doesn’t use grades in hiring and says, “Well, Google proved that grades don’t matter.” Now let’s put aside the fact that we don’t know how Google exactly did this analysis, and whether they actually prove that it doesn’t matter in the Google context. But it’s a pretty big leap to then say, “Because they’ve shown this in one context, that that’s going to be port over exactly to the context that the SVP was thinking about in his company.”

So I think what we have in mind here is just thinking a little bit more about the relevance of findings from one setting to the other, rather than just porting it over exactly or dismissing it all together.

CURT NICKISCH: What’s a good strategy to break out of that when you’re in that situation or when you see it happening?

AMY EDMONDSON: So you can’t see me smiling, but I’m smiling ear to ear because this really falls squarely in my territory because it’s so related to if you want something to be true, it can then be even harder to tell the boss, “Well, hold on here. We don’t really have enough confidence.” So this is really about opening the door to having high quality conversations about what do we know, a really curiosity led conversations. What do we know? What does that tell us? What are we missing? What other tests might we run? And if X or if Y, how might that change our interpretation of what’s going on?

So this is where we want to help people be thoughtful and analytical, but as a team sport, we want managers to think analytically, but we don’t want them to become data scientists. We want them to have better conversations with each other and with their data scientists.

CURT NICKISCH: In teams, as data is being discussed, how as a leader can you communicate the importance of that culture that you’re striving for here? And also how as a manager or as a team member, how can you participate in this and what do you need to be thinking about as you talk through this stuff? Because it’s definitely a process, right?

AMY EDMONDSON: Right. I mean, in a way it starts with framing the situation or the conversation as a learning, problem solving opportunity. And I know that’s obvious, but I have found if that’s not made explicit, especially if there’s a hierarchical relationship in the room, people just tend to code the situation as one where they’re supposed to have answers or they’re supposed to be right. And so just really taking the time, which can be 10 seconds, to specify that, “Wow, this is a really uncertain and fairly high stakes issue for our company, and it’s going to be important for us to have the best possible bet we can.” So what do we know and what are the data telling us and what do we need to learn? And really probing the various people in the room for their perspectives and their interpretations.

So I think starting with that stage setting. And then, like we write about, leaning into questions. We provide a set of sample questions, and they aren’t the only questions or even a cookbook of questions, but they illustrate the kinds of questions that need to be asked. Tone matters. Tone needs to have a feeling of genuine curiosity like, “Ooh, what outcomes were measured?” Not “Well, what outcomes were measured? Were they broad enough?” No, it’s “How broad were they? Did they capture any chance that there were some unintended consequences?” And so forth. So it’s got to be approached in a spirit of genuine learning and problem solving and viewing that as a team sport.

CURT NICKISCH: When can you lean into the answers?

AMY EDMONDSON: There’s never going to be the sort of perfect answer, the crystal ball. There are no crystal balls. So it’s a very good question.

CURT NICKISCH: It seems like to be really good at data-driven decision making, you have to be analytical and you have to have those hard skills. You also have to have the soft skills to be able to lead these discussions among your team and do it in a psychologically safe space. It definitely sounds hard. And you can see why a lot of people go the easy route and say, “Oh, that doesn’t apply to us,” or, “Yes, that’s the gospel truth.” What’s your hope out of all of this?

AMY EDMONDSON: Well, I think my hope is that we all get more comfortable with uncertainty. Start to develop the emotional and cognitive muscles of learning over knowing. Embracing learning, over knowing, and then using the team. This is a team sport. Those are mindset things. And then so that we get more comfortable with a mode of operating that is really just test and iterate, test and iterate. What do we try? What data? What did the data tell us? What should we try next? Life and work in kind of smaller batches rather than these giant decisions and giant roll-outs.

But there’s going to be more navigating the uncertainty, I think, going forward. And we need people who are, as you said, analytical but also curious, also good at listening, also good at leading a team conversation so that you actually can get somewhere. And it doesn’t have to take forever. We can have a conversation that’s quite efficient and quite thoughtful, and we get to a sufficient level of confidence that we feel now we’re able to act on something.

MIKE LUCA: People talk a lot about things like quote unquote “big data” or large scale analytics, and I think there are a lot of interesting innovations happening there. But I also think there are lots of contexts where a little bit of careful data could go a long way. So I think when it comes to many managerial questions, thinking about, is this a causal inference question? And if so, what is the question we’re trying to answer?

From a team perspective, my hope is that people will be focused on trying to answer a question that could then inform a decision. And by thinking about the analytics underlying it and being comfortable with uncertainty, you get to more effective use of data. And that’s both the internal data that’s sitting within your organization, but also the growing amount of external data that’s coming from academic research or news articles and thinking about how to synthesize information from these different sources and then have good group discussions about how to effectively use it.

CURT NICKISCH: Mike and Amy, this has been great. Thanks so much for coming on the show to talk about your research.

AMY EDMONDSON: Thank you.

MIKE LUCA: Thanks.

CURT NICKISCH: That’s Carey School Professor Michael Luca and Harvard Business School professor Amy Edmondson, co-authors of the HBR article “Where Data-Driven Decision-Making Can Go Wrong.”

Want some more data points? We have nearly 1,000 episodes plus more podcasts to help you manage your team, your organization, and your career. Find them at hbr.org/podcasts or search, HBR in Apple Podcast, Spotify, or wherever you listen.

Thanks to our team, senior producer Mary Dooe, associate producer Hannah Bates, audio product manager Ian Fox, and senior production specialist Rob Eckhardt. Thank you for listening to the HBR IdeaCast. We’ll be back on Tuesday with our next episode. I’m Curt Nickisch.