—
We’ve all heard about ‘data-driven decision making’. The past decade has seen a race for more and more data — as if we just had perfect information, we’d make the right decision. This has often been described as being data-rich, but insight-poor.
Taking it to the extreme, it seems paradoxical.
If achieving perfect value-capture from data were the secret sauce to quality decision-making, then in theory, incumbents with coffers of data would make all the right moves. It would be like playing chess against AlphaZero.
Of course, this isn’t the case. Intuitively, we know it’s more than a game of information arbitrage.
In radically uncertain environments, we interpret novel situations, establish ‘what’s going on here’, construct a cohesive narrative with incomplete information, and make decisions.
This is the essence of complexity — it’s adaptive, not optimizing. ‘Adaptive’ means survival in scenarios where the scope of the problem is not well framed, information is imperfect, and the variables change constantly (most of which we can’t observe or measure)
Complexity economics sees the economy as in motion, perpetually “computing” itself — perpetually constructing itself anew. Where equilibrium economics emphasizes order, determinacy, deduction, and stasis, complexity economics emphasizes contingency, indeterminacy, sense-making, and openness to change
The goal in this environment isn’t to somehow know the decision will be right (this is outcome bias), it’s to create the primordial soup for what’s ‘right’ to emerge.
There are far more good ideas you can post-rationalize than pre-rationalize
— Rory Sutherland
Data is an ingredient, but data alone will not drive decision-making without perception and conviction. And conversely, perception and conviction are dangerous and error-prone without data. (unfounded over-confidence)
Cassie Kozyrkov, Chief Decision Scientist at Google, articulates this point well:
Data science without decision science is impotent, just as decision science without data science is impotent.
The concept of ‘decision-driven’ has been thrown around for a little over a decade. Interestingly, an article in HBR about decision-driven organizations popped up in 2010 (written by folks at Bain & Company) — around the same time we see one of McKinsey’s ‘landmark articles’ on the advent of ‘Big Data’.
That same year, Olivier Sibony, who later wrote Noise with Daniel Kahneman, was leading research at McKinsey revealing the importance of dialogue in tandem with analysis.
From there, through the feeding frenzy of big data and analytics, it’s almost a decade of silence until an MIT Sloan article, Leading With Decision-Driven Data Analytics, resurfaces the term.
In the article, Bart de Langhe and Stefano Puntoni broadly define being ‘decision-driven’ as focusing on framing the right questions.
“Data-driven decision-making anchors on available data. This often leads decision makers to focus on the wrong question. Decision-driven data analytics starts from a proper definition of the decision that needs to be made and the data that is needed to make that decision.”
To summarize their principles, being decision-driven is characterized by:
- Framing questions over finding ‘answers’
- Gathering enough information to decide is more important than complete information
- Exploring unknowns instead of optimizing knowns — avoiding pre-mature convergence by going “wide first, then narrow’
- Identifying data blindspots (data that could impact a decision isn’t available)
- Recognizing historical data may not model future events
Similarly, economists Mervyn King and John Kay talk about the dangers of relying on data and models alone when navigating in environments of radical uncertainty where, “all models are wrong, but some are useful”.
They go on to say:
“Models are rarely used as inputs to the decision making process they’re typically used as a justification to a predetermined decision … Data are important, but we should be careful about making inferences, and especially causal inferences, on data alone.”
Mervyn King and John Kay, Radical Uncertainty
King is no stronger to uncertainty — he was instrumental in navigating the financial crisis as the governor of the Bank of England. In his book with John Kay, Radical Uncertainty, they provide multiple examples of when a data-driven approach relied on models as predictors of the future but failed.
They failed to be useful due to a fundamentally flawed assumption with forecasts in these environments — that the future will resemble the past.
In complex systems, we’d be foolish to believe we can account for all (or even most) variables, and as observed through the work of Donella Meadows and others, we can assume that how we think our actions will impact a system is often wrong — or even completely opposite.
A decision-driven approach leans heavily on abductive and inductive reasoning. It relies on trusted, often competing judgments backed by evidence (often different perspectives on the same data) to create a picture of what’s going on, what it means, and bet on the best opportunities to reveal more information — or drive some kind of desired change.
“Successful decision‐makers balance data, experience, and intuition. They quickly sort through information, apply judgment, and are fierce interrogators of data to cultivate sharp insights. They know there is more to decision‐making than just the data. They resist being intoxicated by information.
Instead, they apply first‐order principles to understand what the decision really is, why it must be taken, and to what end. They then seek the relevant data to help make that decision. In short, they make informed decisions with incomplete information.”
Paul F. Magnone, Decisions Over Decimals: Striking the Balance between Intuition and Information
This problem is not about finding answers in the data — the last decade has seen strides in data access, reporting, and discoverability. With AI, answers to questions that used to seemingly require a clunky dashboard will be a prompt away.
The problem seems to be in navigating the unknowns, asking the right questions, and fostering emergence — not chasing targets.
The world cannot be understood without numbers, and it cannot be understood with numbers alone. Love numbers for what they tell you about real lives. — Hans Rosling, Factfulness