Nova: Imagine you are a software engineer at Microsoft back in 2012. You have this tiny idea. You think that changing the way search headings are displayed on Bing might make things a bit clearer for users. It is a small tweak, really, just a minor adjustment to how the font looks. But when you pitch it to your manager, they tell you it is not a priority. It is too small to matter. In most companies, that would be the end of it. Your idea would die right there in that meeting room.

Nova: Exactly. Because Microsoft had built an experimentation platform, this engineer was able to run a simple A/B test without needing a green light from the higher-ups. He launched the test, and within hours, it started showing something incredible. That tiny font change was actually increasing revenue by a massive margin. When they crunched the numbers, that one small experiment ended up being worth over one hundred million dollars in annual revenue.

Nova: It sounds like a fluke, but it is actually the core premise of the book we are diving into today: Experimentation Works by Stefan Thomke. Thomke is a professor at Harvard Business School, and he argues that in the modern business world, our intuition is often dead wrong. The only way to truly know what works is to stop guessing and start testing. Today, we are going to look at how the world's most successful companies have turned experimentation into their primary engine for growth.

Nova: One of the biggest hurdles to innovation that Thomke identifies is something he calls the HiPPO. Have you ever heard that term in a business context?

Nova: Not quite. HiPPO stands for the Highest Paid Person's Opinion. It is that classic corporate dynamic where the person with the biggest paycheck makes the final call on every decision based on their gut feeling or their years of experience.

Nova: That is the traditional view, but Thomke presents some pretty humbling data. He points out that even at companies like Google or Microsoft, where the executives are incredibly smart, only about ten to twenty percent of their ideas actually produce positive results. That means eighty to ninety percent of the time, the expert's intuition is either useless or actually harmful to the business.

Nova: Because for a long time, we did not have a better way. We relied on experience because data was hard to get. But Thomke argues that we are now in an era where the cost of experimentation has plummeted. We do not have to guess if a new feature will work; we can just ask the customers through their behavior. The HiPPO is dangerous because it shuts down the very experiments that could prove them wrong.

Nova: Not exactly fire them, but definitely redefine their role. Instead of being the Chief Decision Maker, the leader becomes the Chief Experimenter. Their job is to build the infrastructure that allows everyone else to test their ideas. It is a shift from a culture of knowing to a culture of learning.

Nova: It is a massive ego hit. But Thomke shows that the companies that embrace this humility are the ones that dominate. When you stop pretending you have all the answers, you open the door to discoveries you never could have imagined.

Nova: If you want to see the gold standard of this philosophy, Thomke spends a lot of time looking at Booking. com. They are arguably the most successful experimentation-led company in the world.

Nova: At any given moment, there are literally thousands of experiments running on that site. Thousands. Almost every single thing you see—the color of a button, the wording of a search result, the placement of a photo—is part of a test. And here is the kicker: almost any employee at the company can launch an experiment without getting permission from a manager.

Nova: Yes, provided they follow the rigorous protocols the company has built. They have a centralized experimentation platform that handles the math and the safety checks. If an experiment starts to tank the conversion rate or break the site, the system automatically shuts it down. But the philosophy is radical decentralization. They believe that the people closest to the product are the ones who have the best ideas, and they should not have to wait six months for a committee to approve a test.

Nova: That is where the infrastructure comes in. Thomke explains that you need a single source of truth. Everyone at Booking. com uses the same metrics to define success. If your experiment improves the primary goal without hurting the secondary goals, it wins. It is not about who has the loudest voice; it is about what the data shows. This allows them to iterate at a speed that their competitors simply cannot match.

Nova: Exactly. And because they test so much, they find those one-hundred-million-dollar font changes that everyone else misses. They understand that most ideas will fail, so the goal is to make the cost of failure as close to zero as possible. If it costs you ten thousand dollars to run a test, you will only run a few. If it costs you ten cents, you will run everything.

Nova: Now, you might be thinking, okay, I will just start A/B testing everything tomorrow. But Thomke warns that it is not just about the tools. He introduces a framework called the 7 S's that a company needs to align to actually make this work.

Nova: It starts with Strategy. You need to know what you are trying to achieve so you are not just testing random things. Then there is System, which is the technical platform like the one Microsoft or Booking uses. You need the Structure to allow for decentralization, and the Skills—meaning your people actually need to understand basic statistics so they do not misinterpret the results.

Nova: Style, Staff, and Shared Values. Style refers to leadership. Leaders have to be willing to be proven wrong. Staff means hiring people who are naturally curious and skeptical. And Shared Values is the big one. The whole organization has to value the truth of the data over the hierarchy of the office.

Nova: You do not need everyone to have a PhD, but you do need a baseline of data literacy. Thomke points out a common pitfall: people seeing a result and assuming it is a success when it is actually just statistical noise. If you do not understand things like sample size or significance levels, you are just flying blind with a fancy dashboard.

Nova: It is. Thomke mentions State Farm as an example of a more traditional company that tried to adopt this. They built an entire innovation center to run experiments on things like insurance claims and customer service. It was a huge shift for a company that is naturally risk-averse, but they realized that the biggest risk was actually not experimenting while the world changed around them.

Nova: One of the most counterintuitive parts of the book is how Thomke talks about failure. In most businesses, failure is something you hide or get punished for. In an experimentation culture, failure is actually a requirement.

Nova: No one likes it, but Thomke makes a distinction between a bad failure and a good failure. A bad failure is when you spend millions on a product launch without testing it and it flops. A good failure is a cheap experiment that proves your hypothesis was wrong before you spent the big money.

Nova: Precisely. Thomke argues that if most of your experiments are succeeding, you are actually doing it wrong. It means you are only testing safe, obvious ideas. You are not pushing the boundaries. To find the real breakthroughs, you have to be willing to strike out a lot.

Nova: That is a huge point, and Thomke does not shy away from it. He discusses the ethical boundaries of experimentation. Remember the Facebook emotion study from a few years ago? They manipulated people's news feeds to see if it affected their mood. There was a massive backlash because users felt like lab rats.

Nova: He suggests that experiments should always aim to improve the customer experience or provide value. If an experiment is deceptive or potentially harmful, it is a no-go. Companies need internal review boards, similar to what universities have, to oversee the ethics of their testing. The goal is to learn how to serve the customer better, not how to exploit them.

Nova: Exactly. And that is why the Shared Values piece of the 7 S's is so important. If the company's value is just profit at any cost, experimentation can become a weapon. But if the value is customer-centricity, it becomes a tool for empathy.

Nova: As we wrap up our look at Experimentation Works, the big takeaway is that the era of the all-knowing visionary leader is ending. In a world that is moving this fast, no one is smart enough to predict the future. The companies that win are the ones that have built a machine for learning.

Nova: Exactly. Whether you are a giant like Amazon or a small startup, the principles are the same: lower the cost of curiosity, empower your people to test their ideas, and have the humility to let the data tell you when you are wrong. Stefan Thomke shows us that innovation is not a lightning bolt of genius; it is a disciplined, rigorous process of trial and error.

Nova: You almost certainly are. And that is how progress happens—one small, measured step at a time. If you want to dive deeper into the frameworks and the math behind this, I highly recommend picking up the book. It is a masterclass in how to build a truly modern organization.

Nova: My pleasure. This is Aibrary. Congratulations on your growth!