Description
Trustworthy Online Controlled Experiments 1st Edition by Ron Kohavi, ISBN-13: 978-1108724265
[PDF eBook eTextbook] – Available Instantly
- Publisher: Cambridge University Press
- Publication date: April 2, 2020
- Edition: 1st
- Language : English
- 290 pages
- ISBN-10: 1108724264
- ISBN-13: 978-1108724265
Getting numbers is easy; getting numbers you can trust is hard. This practical guide by experimentation leaders at Google, LinkedIn, and Microsoft will teach you how to accelerate innovation using trustworthy online controlled experiments, or A/B tests.
Based on practical experiences at companies that each run more than 20,000 controlled experiments a year, the authors share examples, pitfalls, and advice for students and industry professionals getting started with experiments, plus deeper dives into advanced topics for practitioners who want to improve the way they make data-driven decisions. Learn how to
- Use the scientific method to evaluate hypotheses using controlled experiments.
- Define key metrics and ideally an Overall Evaluation Criterion.
- Test for trustworthiness of the results and alert experimentersto violated assumptions.
- Build a scalable platform that lowers the marginal cost of experiments close to zero.
- Avoid pitfalls like carryover effects and Twyman’s law * Understand how statistical issues play out in practice.
Table of Contents:
Half-title
Reviews
Title page
Copyright information
Contents
Preface
Acknowledgments
Part I Introductory Topics for Everyone
1 Introduction and Motivation
Online Controlled Experiments Terminology
Why Experiment? Correlations, Causality, and Trustworthiness
Necessary Ingredients for Running Useful Controlled Experiments
Tenets
Tenet 1: The Organization Wants to Make Data-Driven Decisions and Has Formalized an OEC
Tenet 2: The Organization Is Willing to Invest in the Infrastructure and Tests to Run Controlled Exp
Tenet 3: The Organization Recognizes That It Is Poor at Assessing the Value of Ideas
Improvements over Time
Google Ads Example
Bing Relevance Example
Bing Ads Example
Examples of Interesting Online Controlled Experiments
UI Example: 41 Shades of Blue
Making an Offer at the Right Time
Personalized Recommendations
Speed Matters a LOT
Malware Reduction
Backend Changes
Strategy, Tactics, and Their Relationship to Experiments
Scenario 1: You Have a Business Strategy and You Have a Product with Enough Users to Experiment
Scenario 2: You Have a Product, You Have a Strategy, but the Results Suggest That You Need to Consid
Additional Reading
2 Running and Analyzing Experiments: An End-to-End Example
Setting up the Example
Hypothesis Testing: Establishing Statistical Significance
Designing the Experiment
Running the Experiment and Getting Data
Interpreting the Results
From Results to Decisions
3 Twyman’s Law and Experimentation Trustworthiness
Misinterpretation of the Statistical Results
Lack of Statistical Power
Misinterpreting p-values
Peeking at p-values
Multiple Hypothesis Tests
Confidence Intervals
Threats to Internal Validity
Violations of SUTVA
Survivorship Bias
Intention-to-Treat
Sample Ratio Mismatch (SRM)
Threats to External Validity
Primacy Effects
Novelty Effects
Detecting Primacy and Novelty Effects
Segment Differences
Segmented View of a Metric
Segmented View of the Treatment Effect (Heterogeneous Treatment Effect)
Analysis by Segments Impacted by Treatment Can Mislead
Simpson’s Paradox
Encourage Healthy Skepticism
4 Experimentation Platform and Culture
Experimentation Maturity Models
Leadership
Process
Build vs. Buy
Can an External Platform Provide the Functionality You Need?
What Would the Cost Be to Build Your Own?
What’s the Trajectory of Your Experimentation Needs?
Do You Need to Integrate into Your System’s Configuration and Deployment Methods?
Infrastructure and Tools
Experiment Definition, Set-up, and Management
Experiment Deployment
Experiment Instrumentation
Scaling Experimentation: Digging into Variant Assignment
Single-Layer Method
Concurrent Experiments
Experimentation Analytics
Part II Selected Topics for Everyone
5 Speed Matters: An End-to-End Case Study
Key Assumption: Local Linear Approximation
How to Measure Website Performance
The Slowdown Experiment Design
Impact of Different Page Elements Differs
Extreme Results
6 Organizational Metrics
Metrics Taxonomy
Formulating Metrics: Principles and Techniques
Evaluating Metrics
Evolving Metrics
Additional Resources
SIDEBAR: Guardrail Metrics
SIDEBAR: Gameability
7 Metrics for Experimentation and the Overall Evaluation Criterion
From Business Metrics to Metrics Appropriate for Experimentation
Combining Key Metrics into an OEC
Example: OEC for E-mail at Amazon
Example: OEC for Bing’s Search Engine
Goodhart’s Law, Campbell’s Law, and the Lucas Critique
8 Institutional Memory and Meta-Analysis
What Is Institutional Memory?
Why Is Institutional Memory Useful?
9 Ethics in Controlled Experiments
Background
Risk
Benefits
Provide Choices
Data Collection
Culture and Processes
SIDEBAR: User Identifiers
Part III Complementary and Alternative Techniques to Controlled Experiments
10 Complementary Techniques
The Space of Complementary Techniques
Logs-based Analysis
Human Evaluation
User Experience Research (UER)
Focus Groups
Surveys
External Data
Putting It All Together
11 Observational Causal Studies
When Controlled Experiments Are Not Possible
Designs for Observational Causal Studies
Interrupted Time Series
Interleaved Experiments
Regression Discontinuity Design
Instrumented Variables (IV) and Natural Experiments
Propensity Score Matching
Difference in Differences
Pitfalls
SIDEBAR: Refuted Observational Causal Studies
Part IV Advanced Topics for Building an Experimentation Platform
12 Client-Side Experiments
Differences between Server and Client Side
Difference #1: Release Process
Difference #2: Data Communication between Client and Server
Implications for Experiments
Implication #1: Anticipate Changes Early and Parameterize
Implication #2: Expect a Delayed Logging and Effective Starting Time
Implication #3: Create a Failsafe to Handle Offline or Startup Cases
Implication #4: Triggered Analysis May Need Client-Side Experiment Assignment Tracking
Implication #5: Track Important Guardrails on Device and App Level Health
Implication #6: Monitor Overall App Release through Quasi-experimental Methods
Implication #7: Watch Out for Multiple Devices/Platforms and Interactions between Them
Conclusions
13 Instrumentation
Client-Side vs. Server-Side Instrumentation
Processing Logs from Multiple Sources
Culture of Instrumentation
14 Choosing a Randomization Unit
Randomization Unit and Analysis Unit
User-level Randomization
15 Ramping Experiment Exposure: Trading Off Speed, Quality, and Risk
What Is Ramping?
SQR Ramping Framework
Four Ramp Phases
Ramp Phase One: Pre-MPR
Ramp Phase Two: MPR
Ramp Phase Three: Post-MPR
Ramp Phase Four: Long-Term Holdout or Replication
Post Final Ramp
16 Scaling Experiment Analyses
Data Processing
Data Computation
Results Summary and Visualization
Part V Advanced Topics for Analyzing Experiments
17 The Statistics behind Online Controlled Experiments
Two-Sample t-Test
p-Value and Confidence Interval
Normality Assumption
Type I/II Errors and Power
Bias
Multiple Testing
Fisher’s Meta-analysis
18 Variance Estimation and Improved Sensitivity: Pitfalls and Solutions
Common Pitfalls
Delta vs. Delta %
Ratio Metrics. When Analysis Unit Is Different from Experiment Unit
Outliers
Improving Sensitivity
Variance of Other Statistics
19 The A/A Test
Why A/A Tests?
Example 1: Analysis Unit Differs from Randomization Unit
Example 2: Optimizely Encouraged Stopping When Results Were Statistically Significant
Example 3: Browser Redirects
Example 4: Unequal Percentages
Example 5: Hardware Differences
How to Run A/A Tests
When the A/A Test Fails
20 Triggering for Improved Sensitivity
Examples of Triggering
Example 1: Intentional Partial Exposure
Example 2: Conditional Exposure
Example 3: Coverage Increase
Example 4: Coverage Change
Example 5: Counterfactual Triggering for Machine Learning Models
A Numerical Example (Kohavi, Longbotham et al. 2009)
Optimal and Conservative Triggering
Overall Treatment Effect
Example 1
Example 2
Trustworthy Triggering
Common Pitfalls
Pitfall 1: Experimenting on Tiny Segments That Are Hard to Generalize
Pitfall 2: A Triggered User Is Not Properly Triggered for the Remaining Experiment Duration
Pitfall 3: Performance Impact of Counterfactual Logging
Open Questions
Question 1: Triggering Unit
Question 2: Plotting Metrics over Time
21 Sample Ratio Mismatch and Other Trust-Related Guardrail Metrics
Sample Ratio Mismatch
Scenario 1
Scenario 2
SRM Causes
Debugging SRMs
Other Trust-Related Guardrail Metrics
22 Leakage and Interference between Variants
Examples
Direct Connections
Indirect Connections
Some Practical Solutions
Rule-of-Thumb: Ecosystem Value of an Action
Isolation
Edge-Level Analysis
Detecting and Monitoring Interference
23 Measuring Long-Term Treatment Effects
What Are Long-Term Effects?
Reasons the Treatment Effect May Differ between Short-Term and Long-Term
Why Measure Long-Term Effects?
Long-Running Experiments
Alternative Methods for Long-Running Experiments
Method #1: Cohort Analysis
Method #2: Post-Period Analysis
Method #3: Time-Staggered Treatments
Method #4: Holdback and Reverse Experiment
References
Index
Ron Kohavi is a VP and Technical Fellow at Airbnb. He was previously a Technical Fellow and Corporate VP at Microsoft. Prior to Microsoft, he was the director of data mining and personalization at Amazon.com. He has a PhD in Computer Science for Stanford University. His papers have over 40,000 citations and three of his papers are in the top 1,000 most-cited papers in Computer Science.
Diane Tang is a Google Fellow, with expertise in large-scale data analysis and infrastructure, online controlled experiments, and ads systems. She has an AB from Harvard and MS/PhD from Stanford, and has patents and publications in mobile networking, information visualization, experiment methodology, data infrastructure, and data mining / large data.
Ya Xu heads Data Science and Experimentation at LinkedIn. She has led LinkedIn to become one of the most well-regarded companies when it comes to A/B testing. Before LinkedIn, she worked at Microsoft and received a PhD in Statistics from Stanford University. She is widely regarded as one of the premier scientists, practitioners and thought leaders in the domain of experimentation, with several filed patents and publications. She is also a frequent speaker at top conferences, universities and companies across the country.
What makes us different?
• Instant Download
• Always Competitive Pricing
• 100% Privacy
• FREE Sample Available
• 24-7 LIVE Customer Support
