We come close to the holy grail question of AB-testing.
The question is this: How confident are we in the uplift reported by our analysis?
Indeed, we have a coin and we need to determine if it’s a fair coin or not (a fair coin has 50/50 chance of getting heads or tails). After throwing the coin 3 times we have 2 tails and 1 head. Our analysis shows us a 66.67% rate of tails which is a HUGE indicator that the coin isn’t fair. We had only 3 observations, can we trust the result of this analysis?
When analyzing AB-testing results we could have...