A structured collection of deep-dive learning guides covering statistics, research methods, UX principles, and program management for the other 95%.
A/B testing is a courtroom trial, not a treasure hunt. Most people approach A/B testing like prospecting — dig around in the data until you find gold. That's exactly backwards. An A/B test is a trial. You start with a...
Think of CUPED as noise-cancelling headphones for your A/B test. Your experiment outcome for each user is a mix of two signals: who they *already are* (a power user will always spend more than a casual user) and what...
The courtroom analogy. Think of frequentist and Bayesian testing as two different legal systems for the same crime. The frequentist system works like a strict procedural court: "Assuming the defendant is innocent, how...
Think of your experimentation program as a mutual fund, and the holdout as the audit. Each A/B test is like buying a stock. The analyst says it went up. Great. You buy fifty stocks over a year, each one reportedly a...
Think of a page on your website as a recipe, not a shopping list. A shopping list is additive: eggs are good, butter is good, flour is good, and having all three is exactly as good as the sum of each. If that's how your...
The novelty effect is the sugar rush of product experimentation. When you give a child candy, you get a burst of energy that doesn't represent their baseline activity level. If you measured their "productivity" during...
Think of statistical significance like a metal detector on a beach. The p-value doesn't tell you "there's gold here." It tells you "the detector beeped." Whether there's actually gold depends on how many gold coins are...
The funnel is a supply-side map imposed on demand-side reality. Think of a subway map. It's not geographically accurate — stations that are far apart on the map might be across the street from each other. But the map is...
Friction compounds like interest on debt. Imagine a customer journey with 10 steps. Each step works well 90% of the time. Sounds fine, right? But the probability of ALL ten going well is 0.9 raised to the 10th power —...
Think of an experiment's metrics like flying an aircraft. Your OEC (Overall Evaluation Criterion) is the heading -- the single composite direction you're trying to go. Your guardrail metrics are the engine temperature,...
Survey design is a negotiation between what you want to know and what respondents are willing to give you. Think of respondent attention as a currency with a fixed budget. Every question you ask spends some of that...
Think of your brain as running two departments that don't talk to each other much. Department 1 (fast, automatic, emotional) evolved to keep you alive in small bands on the African savannah: flinch from snakes, follow...
Humans are satisficers with finite bandwidth. Think of every user as a forager in a vast forest. They don't examine every tree. They move *fast enough* (not perfectly), decide *quickly enough* (not thoroughly), and scan...
The researcher is the instrument. In quantitative research, your instrument is a ruler, a scale, a survey with predetermined responses. It measures the same way regardless of who holds it. In qualitative research, *you*...
Segments are probability gradients, not walls. Think of a heat map, not a set of boxes. Roger Martin nailed this: you have a *higher probability* of earning purchases from customers you designed for, and a *lower...
Checkout isn't a pipe. It's a negotiation. The dominant mental model — "the funnel is a pipe, friction narrows it, remove friction to widen it" — is useful but dangerously incomplete. A checkout is a psychological...
A form is not a data collection tool. It's a negotiation. Every form field is a line item in a deal between you and your user. You're asking for something (their data, their time, their trust). They're evaluating what...
A landing page is a continuation of a conversation, not the start of one. Think of it like this: someone overhears you say something interesting at a party (the ad), walks over to you (the click), and you immediately...
You're a hiker trying to find the highest peak in a mountain range you can't see. Quick wins are like walking uphill from where you stand. Every step takes you higher — until you reach the top of your particular hill....
You are not a reporter. You are a translator working between two languages that share words but not meanings. Think of it this way: the word "significant" means "important" in English but "unlikely to be due to chance"...
The Flywheel, not the staircase. Most people picture experimentation maturity as climbing stairs — Level 1, Level 2, Level 3, onward and upward. That mental model is wrong in a way that causes real damage.