{"id":1473,"date":"2025-05-17T02:04:38","date_gmt":"2025-05-17T02:04:38","guid":{"rendered":"https:\/\/tiemensfamily.com\/timoncs\/?p=1473"},"modified":"2025-05-17T02:04:38","modified_gmt":"2025-05-17T02:04:38","slug":"reaction-to-cuped-article","status":"publish","type":"post","link":"https:\/\/tiemensfamily.com\/timoncs\/2025\/05\/17\/reaction-to-cuped-article\/","title":{"rendered":"Reaction to CUPED article"},"content":{"rendered":"\n

On September 15, 2024, Craig Sexauer posted “CUPED Explained<\/a>” where CUPED stands for Controlled-experiment Using Pre-Experiment Data, billed as “one of the most powerful algorithmic tools for increasing the speed and accuracy of experimentation programs.”<\/p>\n\n\n\n

Here is a list of some of the things that just make me want to vomit about this explanation of the tool (if not the actual tool itself):<\/p>\n\n\n\n

As an experiment matures and hits its target date for readout, it\u2019s not uncommon to see a result that seems to be\u00a0only barely<\/strong><\/em>\u00a0outside the range where it would be treated as statistically significant.\u00a0<\/p><\/blockquote>\n\n\n\n

“Only barely<\/strong>“. As in, we already gave you a 1-in-20 chance of not proving your hypothesis, you just need a little bit more. This is very reminiscent of this XKCD comic<\/a> showing that green jelly beans linked to acne, where “only 5% chance of coincidence” prominently displayed, while burying the other Red, Turquoise, Magenta, Yellow, Grey, etc (19 other times) studies. Any study that admits to using CUPED would instantly get a “what are you not telling me?” rejection.<\/p>\n\n\n\n

Waiting for more samples delays your ability to make an informed decision, and it doesn\u2019t guarantee you\u2019ll observe a statistically significant result when there is a real effect.<\/p><\/blockquote>\n\n\n\n

See above – you have a weak hypothesis, and adding more sample doesn’t guarantee the result you want. This is all true – it just hides the “use CUPED to solve this problem” conclusion.<\/p>\n\n\n\n

Conceptually, if one group has a\u00a0faster average baseline<\/strong>, their experiment results will also be faster. When we apply a CUPED correction, the\u00a0faster group\u2019s metric will be adjusted downwards relative to the slower group.<\/strong><\/p><\/blockquote>\n\n\n\n

Before you just pick a random variable and “correct” for it (like, by using CUPED), I would recommend you read (and understand) The Book of Why<\/a> by Judea Pearl. This book has tons of examples where a “confounding variable” is corrected for incorrectly. The book describes p-calculus as a way to model causality [All of this is not to say I understand p-calculus. I only understand enough to tell when other people are wrong.]<\/p>\n\n\n\n

Python libraries on causaility:<\/p>\n\n\n\n