An interactive essay · Adapted from the author’s 2014 essay · Read the original essay
A friend once described an RPG bot. It reads no game memory and knows no damage formulas. It watches one pixel on the screen. When that pixel turns from red to gray, it presses heal.
It never tries to understand the whole game. Same problem here: analysis can run forever, but we still have to come back with a result.
Move your pointer up or down. On a phone, tap, or drag the ↕ handle on the right. Each deeper level brings more answers—and twice as many new questions.
Related interactive essays: Problem Solving and Design · Designing Design
Watch the color of one point on the HP bar. When it turns from red to gray, health has fallen below a threshold: press heal. When it changes the other way, do nothing.
The game’s internal logic is a black box. But to be playable, it has to leave input and output interfaces for the human, so that person and machine together make up a complete game system. The bot exploits exactly this: take a small piece of output, apply one simple rule, and take over a tiny loop from a person’s eyes, brain and hands.
Watch whether a heavy hit can cross the probe.
if pixel: red → not red press healelse do nothing
The bot has no ambition to decode the whole system. It grasps just a little of the output. That frees the person from this small, simple loop to put their abilities into something more complex and more interesting. It is a textbook case of modular thinking.
Try “Poison the hero.” The health bar turns green, but the bot only knows red and gray. It can no longer detect that red-to-gray transition. Its tiny piece of understanding works only within its boundaries. (The poison scenario was added for this interactive edition.)
In data science, the analysis we do on data can be divided into types by its goal, and the difficulty varies accordingly. The easiest is descriptive: only describe the data you see. The hardest is mechanistic: fully understand the system that produced the data—exactly which variables changed in each individual, and exactly what other changes those caused.
Take a pear. Work down the levels to see how the questions deepen and how much more you need to answer them.
Enlarge the diagram for small labels, then swipe sideways to explore.
The original essay described only the two ends. Description asks me to record a pear’s shape, color and size. Mechanism asks why it has that shape, color and size, and how the three relate. Then comes another question: if we try a different pear, does our model still hold?
The six types come from Jeff Leek’s Types of Questions lecture in The Data Scientist’s Toolbox on Coursera. The four middle pear examples and the variables in the mechanism diagram were added for this edition. They are illustrations, not botanical findings.
Ideally we would want results at the level of mechanistic analysis: to pick out, clearly and precisely, the flap of one butterfly’s wings inside a hurricane. In practice, though, the information we can get hold of is always far too little.
Below are two identical worlds, with one difference: in one of them, a butterfly flaps its wings once.
An illustration, not a weather forecast. The hurricane is steered by Lorenz’s 1963 three-variable convection model (σ=10, ρ=28, β=8/3, fixed-step RK4, so every device computes exactly the same result). Which “wing” of the attractor x is on stands in for the steering pattern that pushes the storm south or north; one model time unit is drawn as one day, for illustration only. The largest Lyapunov exponent of this system is about 0.906, so the gap doubles on average about every 0.77 days. In the real atmosphere, small-scale errors also cascade up to larger scales, which led Lorenz (1969) to suggest that day-to-day weather has a predictability limit of about two weeks. Nor is the butterfly the “cause” of the landfall: the title of Lorenz’s 1972 talk was itself a question, “Predictability: Does the Flap of a Butterfly’s Wings in Brazil Set Off a Tornado in Texas?”
And what about now, in an age that produces enormous amounts of data? There is more data, certainly, but extracting information from it has become harder too. In this model, raising the number of observation points from 12 to 9,000 cuts error about 27-fold, yet stretches the useful forecast only from about 7.4 to about 11.1 days, a gain of 3.7 days: each e-fold (about 2.7×) cut in error buys only about 1.1 days. Detecting the butterfly’s difference takes until day 9.2 with 12 points and still until day 6.1 with 9,000, while the hurricanes part ways around day 14.5. At times like this we have to rein in our greed and advance one step at a time.
Sometimes, having weighed cost against return, we decide that studying a problem to a certain depth and breadth already meets our needs, as with that bot. Then we should spend the remaining resources where they count.
The three problems below share one budget of time and effort. Each column runs down the same six levels of analysis. Choose a cell to decide how far to go.
Each bar shows the additional value that level brings back. The number at the lower right is the time and effort it costs.
To analyse a problem system well, you certainly need the focus and persistence to get to the bottom of it. But we have to keep reminding ourselves that in the rough real world, the problem we are solving must have a boundary, one that keeps us from losing the way back while we dig our tunnel.
The first rule of analysis:
come back with a finite result.
This is a limit we cannot get around.