This chapter on AB-test analysis gives you a head start into running and analyzing AB-tests. It goes way beyond “which button has a higher CTR” approach and gives you tools to holistically analyze AB-tests.
AB-testing is a tool to learn about our users behavior, test new designs, features, it’s many things.
In short, during an AB-test users are split into cohorts and each cohort sees their unique AB-test variation – a completely different UX. Later, we look at the performance of each variation and trying to find a winner – a variation that later will stay in the app for all users.
AB-test analysis requires a certain mindset. Instead of blindly comparing a CTR or a single metric to find a winning variation, I highly recommend to learn why it is the case.
In other words, I recommend planning every AB-test as a proper behaviorial experiment. Start by forming a hypothesis in the beginning and then use Data Analysis to either confirm or deny this hypothesis.
This is where Funnel Analysis comes handy – we can compare an entire AARRR funnel for every variation and see its effects. This is exactly what you’ll be doing in this chapter, let’s go!
Hi, it’s Anatoli, the author of SQL Habit.
SQL Habit is a course (or, as some of the students say, “business simulator”). It’s based on a story of a fictional startup called Bindle. You’ll play a role of their Data Analyst and solve real-life challenges from Business, Marketing, and Product Management.
SQL Habit course is made of 13 chapters (you’re looking at one atm) that contain 271 bite-sized lessons and exercises. All of them have a real-life setting and detailed explanations. You can immediately apply everything you’ve learned at work.
The 2nd part of the course is called Practice. It’s made of standalone exercises based on multiple datasets – E-commerce, Finance and Meditation app a-la Headspace or Calm. Practice exercises are harder than in the main course. They’ll get you ready for any challenge at work or an interview.