Thinking in experiments: Today’s winning mindset

[unable to retrieve full-text content]

The old excuses for not engaging with customers more often simply don’t apply anymore. In fact they’ve been turned on their heads.

Thinking in experiments: Today’s winning mindset. Article by Scott McDonald
Photo by Girl with red hat on Unsplash

“For every good idea there are a thousand bad ideas it is indistinguishable from. The only real way to tell the difference is to go out and try them, see what works, discard the failures and build on the successes. You have to, in other words, dare to be crap.”

— Marc Randolph, Co-founder, Netflix

Perhaps Randolph was a bit dramatic, but he makes an important point. We have to wade through a lot of crap ideas to identify those relative few that will grow our businesses and secure our futures.

Yet it’s still surprisingly common today for companies to bet big on launching a few cherished ideas into the marketplace each year, with little in the way of pre-market testing.

We don’t have to do that anymore, and we can’t afford to either. Here’s why…

The innovation deck is stacked heavily against us

A growing number of studies show that only about 10% of new business ideas will find some success in the marketplace, and just one in 250 will hit it big ($1+ billion). So despite the mythology and romance of the “big idea” in business, innovation today is as much a process of elimination as it is of inspiration. And all that elimination needs to happen quickly. So fortunately…

The opportunity cost of testing new ideas is plummeting

It has become ridiculously cheap and easy to test our ideas in the marketplace. Frameworks are maturing. Research platforms are more powerful than ever and a growing array of no-code tools are giving non-developers the ability to quickly prototype all sorts of applications on their own. Today you can drag and drop an e-commerce website into existence using an app on your phone.

Where we once rushed to launch, we can now rush to learn. Experimentation as a core business practice and leadership philosophy is exploding.

  • We have a new generation of powerful research platforms like Feedback Loop, Suzy and Remesh that allow teams to run a variety of surveys and experiments with greater speed and scale than ever before.
  • There’s a new breed of consulting firm like Exponentially, LeanApps and Precoil that test new ideas for organizations, or teach them how to run experiments themselves.
  • There’s even a movement afoot to make experimentation skills part of the core curriculum of MBA programs. See Why Business Schools Need to Teach Experimentation.
David Bland on Twitter: “I don’t think you all need help coming up with more ideas. You need help testing those ideas.”

Look before you leap

No matter what your challenge, customer, product or market, there are almost always ways to for test product-market fit early.

  • A new line of cosmetics? Test the packaging first.
  • A new service offering? Create a mock brochure or sales deck and start selling now.
  • New consumer product? Create a landing page and run some inexpensive ads.

Jeff Hawkins famously tested his Palm Pilot idea by carrying around a piece of wood with drawn-on buttons. Just get your idea out there fast, listen carefully and above all don’t get too attached to it.

“If you double the number of experiments you do per year you’re going to double your inventiveness.”

` — Jeff Bezos

What can go wrong when we don’t test early and often? Consider the case of streaming startup Quibi.

The $1 billion learning experience

Quibi (short for “quick bites”) seemingly had it all. A-list co-founders in Jeffrey Katzenberg and Meg Whitman. $1.7 billion in funding from investors like Disney, 21st Century Fox, NBCUniversal, Sony Pictures, Viacom and Alibaba and a level of buzz most startups can only dream about.

Yet six months after launching in April of 2020 and having spent over $1 billion, the founders announced they were shutting the service down after failing to gain traction with subscribers.

“We had a new product and we asked people to pay for it before they actually understood what it was. I think we thought there would be easier adoption,” Katzenberg said. “In the end, we didn’t get the support of consumers and customers in the way we had to to make this a successful business.”

The company did run experiments. Only too late in the game. “Over the summer we started to see a slowdown in our momentum, and we tried many different things, different packaging models, we changed our marketing, we changed the app around many different times but it was clear that for whatever reason this wasn’t going to be as successful as Jeffrey and I had hoped,” Whitman told CNBC at the time.

“Failure is part of the Innovation game, but expensive failure is not a necessary part of the process.”

— Tendayi Viki

Contrast Quibi with Farmers Insurance and its new platform aimed at millennials, Toggle.

Farmers, a Fortune 500 company in a highly regulated industry, worked with the rapid consumer insights platform Feedback Loop (formerly Alpha) to launch a new online business 13 months ahead of plan. In that time the team covered more research ground than most leaders might think possible:

  • 175 surveys and experiments conducted in nine months.
  • 54,000 research participants.
  • 40 audience segments explored.
  • Reduced the time from ideation to product launch from a projected 18 months to five months.

The old excuses for not engaging with our customers more often (no time, no budget) simply don’t apply anymore. In fact they’ve been turned on their heads.

Vision or hallucination?

There’s only one way to find out.

Recommended Reading


Thinking in experiments: Today’s winning mindset was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.

Leave a comment

Your email address will not be published.