More than ever before, managers are using large-scale randomized controlled trials (i.e., experiments) to guide decisions. This has led to impressive gains for organizations ranging from Amazon to the UK government. We are excited about the rise of experiments in organizations and have spent much of the past few years thinking about how to design and interpret them. At the same time, we’ve seen that experimentation remains uneven across and within organizations, and many companies struggle with ways to start or expand experimenting.
Increasingly, companies are using experiments to guide them in their decision making—but many are still missing opportunities, or are failing to implement experiments well. When it comes to the rollout of new products, one particularly effective new kind of experiment involves randomizing the introduction of new products across a set of markets. Uber used this strategy before rolling out its Express Pool service, and Airbnb did the same before rollout out a new landing-page design. In both cases, the companies gathered data that allowed them to roll out their products with confidence that they would succeed—as indeed they did. Many companies, even those not in the tech sector, can benefit from this kind of experimentation, especially if they follow a few basic guidelines.