Stefan Thomke is a Harvard professor and author of the new book Experimentation Works: The Surprising Power of Business Experiments. We talk about why experimentation matters, how to overcome the fear of failure and some of the latest trends that are driving companies to include a rigorous experimentation process into their business.
Hey listeners, before we start this week’s episode, I wanted to let you know that we recorded a number of interviews before the Corona virus disruptions started, wanting to give some context before we jump into some of these shows. Thank you very much for listening, being part of the Inside Outside Innovation community, we look forward to talking more about the disruption of the Corona virus and other things.
Inside Outside innovation is the podcast that brings you the best and the brightest in the world of startups and innovation. I’m your host, Brian Ardinger, founder of InsideOutside.IO, a provider of research events and consultant services that help innovators and entrepreneurs build better products, launch new ideas, and compete in a world of change and disruption. Each week we’ll give you a front row seat to the latest thinking tools, tactics, and trends and collaborative innovation. Let’s get started.
Interview Transcript
Brian Ardinger: Welcome to another episode of Inside Outside Innovation. I’m your host Brian Ardinger, and as always, we have another amazing guest. Today we have Stefan Thomke. He is a Harvard professor and author of a new book that just came out called Experimentation Works: The Surprising Power of Business Experiments. Welcome to the show.
Stefan Thomke: Thanks Brian.
Brian Ardinger: I’m excited to have you on the show because obviously in this corporate innovation space and even in the startup innovation space. Experimentation has gotten a lot more buzz as people try to understand how to navigate the world of uncertainty. I wanted to start the conversation by talking a little bit about why is experimentation so important.
Stefan Thomke: Brian, I mean that’s a great question, and in fact, it’s the uncertainty that makes experimentation so valuable. If you think about uncertainty, you can think about different types of uncertainties that companies face every single day. At one level there’s R & D uncertainty. You know, I’ve had the pleasure of working with lots of R & D organizations over the years. I’ve been at this for more than 25 years. And there the question is, it could be a product, a service, or a new customer experience, does it work as intended?
Another set of uncertainties is what I call scale up uncertainty. If I’m sitting on a scale upside where I have to scale up the service or scale of production, I worry about a different set of questions. I worry about whether something can be effectively made or scaled up. You know, worry about can it be done at high quality, low cost, large volume, and so forth. And if I’m customer facing, I worry about yet another set of questions. And that is, does anybody want it? If they say they want it, do they really mean it and are they willing to pay for it.
And then finally, if I’m running a business unit, of course, I need to make an investment decision, and the question here or the uncertainty here is the opportunity big enough? Does it justify the resource investment? And the problem, of course here is that the tools that we have, like, you know, calculating an ROI on net present value and also all these kinds of wonderful tools, they start breaking down, when you’re dealing with a lot of uncertainty, when something is really novel, you know, how do you put a net present value on something that doesn’t exist yet?
And so these are the sets of uncertainties that we face every single day. My argument is that experimentation is really the best way to address it because it gives me information about cause and effect, which a lot of the other ways of approaching this problem don’t do.
Brian Ardinger: Experimentation…A lot of people think about it as the scientific method and get scared from it. A lot of folks in business world are not necessarily scientists and that, talk a little bit about what are the major barriers to business folks understanding what experimentation means and then how to adapt that.
Stefan Thomke: That makes sense to do a quick detour and ask ourselves, what do we really mean by experiments? And let me tell you what I don’t mean often when people talk about experiments, what they’re really saying is, I’ve tried something. It’s often the way you use it in the English language or in companies what I’ve always run into is we’ve tried something, and it didn’t work and therefore it must’ve been an experiment. Right? That’s not really what I mean.
I mean, a much more disciplined approach, like the scientific method, which by the way was essentially conceived 400 years ago, almost exactly 400 years ago in 1620 by Francis Bacon when he wrote the book Novum Organum, which was a new instrument for building an organizing knowledge. Now, it was done for science back then, but my argument, it’s the same thing for knowledge about management and knowledge about behaviors and so forth. The experiment is at the heart of actually finding or building this new knowledge and organizing knowledge.
Now, what is an experiment? Let me give you the gold standard. And then we can work from there. In an ideal experiment, you’ve got to test. And what you want to do in an ideal experiment is you want to separate what we call independent variables. This is the presumed cost, and it’s something that you’re trying to change from a dependent variable, which is the observed effect, while holding everything else, all the other potential changes constant.
Here’s an example, so imagine you’ve got a sales force. And you’re coming in and you want to give them a bonus. The independent variable is the bonus. And the dependent variable, the observed effect, would be a lift in sales, for example. And what I want to do is I want to understand whether one causes the other to happen, so causality again. Without having the experiment polluted by a lot of other things that are changing. For example, you know, whether the salesperson doesn’t appeal well that day or so on and so on.
So that’s really the gold standard that we’re after, and in a real-world experiment as opposed to science where I can create a laboratory where I can control a lot of these other things. We randomly distribute all the other things that could influence the experiments evenly across the people that we’re testing on. And we want to do is of course blind, so we don’t know who we’re experimenting on and they don’t know that extra, they’re being experiment on. But ff course, in the real world, there are some limitations in terms of what we can and cannot do. The hypothesis, of course, is at the heart of this, and your listeners may remember that the scientific method maybe from high school science days.
Brian Ardinger: Talk a little bit about how companies can start both identifying what to experiment on and the process to begin putting that into their culture.
Stefan Thomke: The process of beginning putting that into their culture begins with an awareness that experimentation matters. That this is really important for the reasons that we just discussed. You know, the uncertainty. Once you’re aware and you understand sort of that in fact, experiments or experimentation is the engine of innovation. You want to then think about what kind of framework do I adopt? You want to have a rigorous framework around it. A little bit light on the scientific method, but that the other factors, in a sense of what makes a good experiment. And in fact, Brian, I have a whole chapter dedicated to this one question, what makes a good experiment.
And then from there, once you decide that we’ve got a good framework, we’ve got the tools in place, then the next step up would be to commit to it, and that is allocate the resources, change the organization. There are actually organizational questions that you need to address, like who owns, for example, this experimentation capability. Then once you’ve got that in place, you need to diffuse it, widen the scope. We give people access to tools, and then you’ve reached the final stage, which I call embeddedness. It’s about democratizing experimentation. That is anybody in the company ought to be able to run an experiment without red tape or running it up, sort of the corporate ladder. And so, you’ve got awareness, belief, commitment, diffusion, embeddedness, failure.
Brian Ardinger: This concept of fear and failure where executives conflate failure with incompetence, versus the idea of a failure of experiment may be just the fact that your assumptions didn’t hold true to what your original thoughts or beliefs were. And so creating a culture that allows that and an opportunity to really provide the executive or the team an opportunity to have the market or data help make those decisions, oftentimes makes it easier for teams to take away the biases and take away the politicalness of a lot of these decisions because you’re letting the data and the results help drive that.
Stefan Thomke: Absolutely Brian, and that’s a word in the community before that is called the hippos, right? The highest paid person’s opinion. Anybody know that? Hippo is a very dangerous animal. Nothing stalls innovation as a powerful hippo. Now, the way I usually think about failure is in a somewhat of an interesting way. I draw a distinction between what I call failure and mistakes. Even though they’re semantically very close to each other. And a mistake for me something that produces no useful information because there is no question that you’re trying to answer here. Like an operational execution, like if Amazon builds yet another distribution center. There is no real question that is being answered here.
Failure on the other hand, usually proceeded by a question. That’s why failure is so important to innovation because in innovation, we’re trying to answer questions. Good failure, that’s absolutely sort of essential to this because it allows us to make progress. There’s a great story. I don’t know if you’ve heard this before, Brian. Old Watson Senior, who at the time was chairman of IBM…Young manager at IBM had just loaned $10 million and was asked to come into Mr Watson’s office and tell him what happened.
And the young manager walks into the office, then of course, he’s shaking his shoes, and before Watson could say anything, that young manager says, Mr. Watson, I take full responsibility for losing $10 million for IBM, which was even then a lot of money, and he was ready to resign, and Mr. Watson looked at him and said, young man, have you lost your mind? I just spent $10 million on your education. Why would I want you to leave?
The point here is that Mr. Watson, even way back then understood that success is not about getting it right the first time. Success is about iterating fast. I call it a high velocity experiment. The ability to iterate fast, to learn from this, to try again, to have another hypothesis, and we kind of work through this as quickly as we can and success is about getting there first before anybody else, and that involves experimentation driven approach to solving innovation problems.
Brian Ardinger: Can you talk a little bit about how to prioritize which experiments to run or quite frankly, even what to experiment with?
Stefan Thomke: That’s a really great question. That’s a question that I get a lot. The problem, of course with experiments is that at the outset you don’t know what happens. Otherwise it wouldn’t be an experiment. And so sometimes experiments really surprise you. I mean, this is what you’re looking for. Now it helps though to have some sense of direction. When I talk about the role of senior leaders in organizations, one of the roles of senior leaders is to set a grand challenge.
Some big goals, such as we want to be number one in customer experience in our industry, or something like that. And that grand challenge then ought to be broken down into testable hypotheses and performance metrics. When you think about what kinds of experiments to run and what not to run. That should give people some sort of guidance, some direction in terms of where the experiment should head. Otherwise people are just running these things Willy nilly without any sense of direction. So that then becomes part of an experimentation program.
Then of course you have different kinds of experiments, Brian. The other kinds of experiments that optimize something like for example, a landing page. If you go to Amazon, for example, or to a booking.com or any of these companies, everything that you see on these landing pages Brian, has been optimized for experiments. And in fact, these companies run tens of thousands of experiments, live experiments on you and me, every year. And everything has been optimized. You have these kinds of experiments moving the position of a button, the size of the font, the color, all of these kinds of things, and always have the same objective. And that is to create stickiness on their websites and get us to convert.
Brian Ardinger: Obviously with the Amazon or booking.com whether they’re getting thousands and thousands of customers that are coming to their websites and they can very easily A/B test and they’ve got enough throughput and people to experiment with. Are there things that like a startup or company that’s just getting started on this, are there best practices to focus on when they’re starting to develop these experiments.
Stefan Thomke: First of all, let me remind everybody that even these big companies, that we talk about now, they all started small. And at some point, an Amazon and a booking.com, they were all startups. And in fact, when you look at what they did, even as startups, they started out with experiments. I mean, they had built their capabilities gradually over time. And you’re listening to Bezos, for example, and he’s been talking about experiments almost from day one. Just read the annual shareholder letter that he personally writes, and he’s been talking about experiments forever. He understands this, you know, even a booking.com started early on. They have a lot of traffic and that helps. They can run a lot more experiments, but you don’t always need a lot of traffic to run experiments. You can actually work with a lot less.
There is of course, something in statistics called the power of an experiment, and so you need to have enough sample size to power an experiment, but it turns out that is influence. The sample size is influenced by, for example, the size of the change. It turns out that if you make bigger changes, you need smaller sample size. And if you make very small changes like the color of a font or something like that, you need bigger sample sizes. And the intuition here is you’ve got a lot of background noise, right? There’s stuff going on in the background and you’ve got a signal, which is the change that you make, and you want to have a sample size that’s big enough that allows you to distinguish the signal from the noise.
There are also lots of experiments that are run in brick and mortar. And brick and mortar of course, you’re dealing with very small sample sizes. Sometimes a small sample sizes are only as big as 50 or even a hundred or so, and there, you can use big data methods, others, algorithmic solutions and things like that, increase the power of the tests. Given a smaller sample size, for example, focusing on better controls.
There are lots of methods around Brian, that can help you to operate even in an environment where you sample sizes are smaller. Described them in the book as well. What you can do if you’re, for example, sitting in a startup or so. It turns out that, by the way, Brian, and a colleague of mine has looked at this in a one form of experiment, A/B tests. You know, it has been adopted quite a bit in startups. In fact, they’ve been all over it.
And one of the reasons I think why they’re adopting, and it’s first of all, the tools are fairly inexpensive nowadays, so you can get good tools. They do have enough traffic that helps as well, and it actually helps them to save on a lot of the marketing costs, market research costs. It turns out that market research is really expensive, so what they end up doing is why not just run the experiment? Why waste all that effort on market research and focus groups and all these kinds of things that don’t work often anyway.
Brian Ardinger: I like that point too. We oftentimes think that we need a data scientist and a person on staff that can help us create these experiments, and obviously there are problems if you create the wrong experiment or do surveys, you can get bad results in that, depending on how you develop surveys. But like you said, there’s a variety of different tools out there that are making it more accessible for folks to start experimenting. Tools out there, help the person walk through designing an experiment that can work and be scientifically valid and things along those lines. Are you seeing any other trends out there that are making it easier for folks to make experimentation more important in their company?
Stefan Thomke: Digital businesses are growing even for non-digital businesses. The digital component is growing as well, we just get a lot more traffic. And when we get more traffic, we can run more experiments on our customers and also, we’re more direct, which is another advantage. It’s not so easy to run a good experiment, say in our retail store, because we’re often not so direct, and sometimes they’re intermediaries and all that.
But the beauty, of course, in online space is that we have a direct interface to our customers, and they can respond very quickly, so we can pick up the effect on our customers almost right away. There’s a lot of trends that we’re seeing that go in that direction. And in fact, and I’ve talked to a number of CEOs about this, and the ones who really jumped on the bandwagon, got it. I mean other than, you know, the Bezos’s of this world. They tell me and they said, listen, in an increasingly digital world, you have to do this. If you don’t do large scale experiments, you’re going to be dead.
I think the time has come here where experimentation is not something that just scientists do now, or engineers do. This is already an important corporate capability that gives you a competitive advantage. And the reality is by these big companies that are super successful, that at one point were startups, they all do this at massive scale. Pretty much everybody on, who’s listening right now, has been on these platforms in the last 48 hours. Well, if you were, you are part of their experimentation ecosystem, whether you like or not. They’re running things all the time and that has given them a big edge over some of their competitors.
Brian Ardinger: Well, I encourage people to pick up this book called experimentation works. If people want to find out more about yourself or about the book, what’s the best way to do that?
Stefan Thomke: I have a website, you know www.THOMKE.com, which takes you directly to my Harvard Business School website. Send me an email T, just the letter t@hbs.edu for HBS for Harvard business school. You can send me something on LinkedIn. Just tell me that you were listening to Inside Outside Innovation, so I know where the connection comes from. And there’s also a new Harvard Business Review issue, just came out. On how to design an innovation culture. So that’s going to be all over the place. This is a great time for experimentation right now, and I invite everybody to be part of the movement.
Brian Ardinger: I appreciate you coming on Inside Outside Innovation to share your insights about what the new world of experimentation is all about. And I appreciate you being out there and fighting the good fight around that. Thank you very much for being on the show.
Stefan Thomke: Thanks for having me.
Brian Ardinger: That’s it for another episode of Inside Outside Innovation. If you want to learn more about our team, our content, our services, check out InsideOutside.io or follow us on Twitter @theIOpodcast or @Ardinger. Until next time, go out and innovate.
FREE INNOVATION NEWSLETTER
Get the latest episodes of the Inside Outside Innovation podcast, in addition to thought leadership in the form of blogs, innovation resources, videos, and invitations to exclusive events. SUBSCRIBE HERE
This post contains affiliate links that may earn Inside Outside a small fee on purchases originating from them. They do not influence editorial decisions to include mention of any products or services in this article or add any cost to the customer.
