Square watermelons have been named one of Japan's greatest innovations, a surprising solution to customer problems, a way to export more watermelons in shipping containers, and a better watermelon that fits better in the refrigerator. But they are bland and ultimately not

I'm teaching my first workshop/mini-course on advanced topics in practical A/B testing at EXL + Unite 2024 on October 9th in Austin, TX. The 3-hour workshop covers 10 topics shown at bit.ly/EXL2024Advance… This is an advanced course, assuming you know the basics of A/B

Goodhart’s Law with Examples at linkedin.com/pulse/goodhart… The short article has three real examples of how organizations fooled themselves with watermelon metrics: green on the outside, but rotten red on the inside. Setting the Overall Evaluation Criterion (OEC) for #abtests is

Slides for my KDD talk today on False Positives in A/B Tests: bit.ly/falsePositives… The paper is at bit.ly/FalsePositiveI… #abtest #experimentguide #successRate

A Quick Introduction to A/B Testing: A 30-minute live Maven 🏛 Lightning Lesson Oct 17 at 8AM PDT. Sign up (free) at bit.ly/QuickIntroABT Taught by best-selling co-author of Trustworthy Online Controlled Experiments : A Practical Guide to A/B Testing

Honored to see my Maven 🏛 course selected as Top course in Growth by Lenny Rachitsky. Lenny's list: maven.com/lenny?utm_sour… Last year, Lenny did a podcast with me that was viewed over 40,000 times on YouTube: bit.ly/ABTestingGuide… Want a quick taste of A/B testing? On Oct 17 at

New online course: Advanced Topics in Practical AB Testing: bit.ly/AdvancedABRKX Two sessions of two hours each on Dec 16 and 18. Following the sold-out workshop at #EXLUNITE (bit.ly/EXL2024Advance…) and requests to teach this online, I just scheduled this class. For

Capping metrics is an often-overlooked trivial technique to increase your statistical power of an #ABTest. If you have a metric like revenue/user, any value over the cap is simply set to the cap. Because revenue is highly skewed, the extreme values increase the variance

The Optimizer’s Curse, or the Winner’s Curse. When we run A/B tests and choose statistically significant results, our estimates are biased by 13% even if perfectly run with no other biases and 80% power (lnkd.in/gsaYve6J). The bias grows when we pick the best variant in

A/B testing without sufficient users is like attempting a marathon without appropriate training. -  Don't run a marathon without first building up your weekly mileage to 20-30 miles. - Don't fly a plane solo without first logging significant hours with an instructor to master

Should you run A/A/B tests with 1/3 of users in each variant? This is a common question I hear, and the short answer is no—this approach reduces statistical power. The optimal allocation (maximum power) for an A/B test is uniform, that is 50/50%.  Deviating from this, such as in

Upcoming cohorts of my A/B Testing courses on Maven are now open for registration: - Mar 3-13: Accelerating Innovation with A/B Testing: bit.ly/ABClassRKT, rated 4.7/5 Recent feedback: bit.ly/ABCourseReviews - Mar 17-20: Advanced Topics in Practical A/B Testing:

A 30-minute + Q&A live Maven 🏛 Lightning Lesson Feb 20 at 8AM PST: A/B Testing Myths Sign up (free) at bit.ly/ABTestingMythsX We will cover myths voted highest, including: - Doing <X> improved revenue by 50% - Running concurrent experiments is invalid due to interactions -

Why you should default to running #ABTests at 50/50? You sometimes hear: this is a risky idea, so we should run it at 10%, or this is a learning experiment that’s expensive (e.g., removing ads), so let’s run it at 10%. The logic for running a treatment at 10% is often flawed.

The next cohort of Accelerating Innovation with A/B Testing with Maven starts May 12, 2025. Sign up at bit.ly/ABClassRKT The logos slide shown, of companies that sent at least two people, has been updated and now includes Wikimedia, Splunk, Estee Lauder, Docusign, StockX,

The "Student's t-test," widely used in A/B testing, could have been called the "Guinness t-test,” if it wasn’t for management's reluctance to allow publications associating the company’s employees with the published research. William Sealy Gosset (1876–1937) developed this

Honored to be listed in AB Tasty's 16 Experimentation Influencers You Should Follow: bit.ly/44g7nuK As for the image background color, I think it should be A/B tested :-) My online interactive course: Accelerating Innovation with A/B Testing starts this Monday, July

Next Monday, 8 Sept 2025, we start a new live cohort of the course Accelerating Innovation with A/B Testing on Maven. Register at bit.ly/ABClassRKT Five x 2.5-hour interactive sessions. Maven rating 4.7/5.0. Most employers reimburse the course, given the high ROI. Human

Running concurrent A/B tests is essential to scale. Many organizations hesitate to run experiments in parallel, fearing that they will interact, but the concerns are overstated: - Concurrent testing is essential to scale - Strong interactions are rare in practice - Most