A/B testing bullet point简洁总结

这是一个很general的a/b testing process总结，个人觉得适合面试之前过一遍，加深印象。楼主最近木有面试，只能默默的准备，希望大家都能stay positive，早上拿到offer。


A/B test:
Understand what drives business and make data informed business decisions.
To understand the causal relationship not simple the correlations.

When to run A/B test:
•        Decide whether or not to launch a new product or feature
•        Quantify the impact of a feature of product
•        Compare data with intuition(to understand how user response to certain parts of a product.

When not to run A/B test:
•        No clear comparison bw control and experimental group
•        Emotional changes needs time: Logo/brand
•        Response data hard to gather(1 year)
•        Too long to response

10 steps guide to setting up an experiment
1.        Define your goal and  form hypo
2.        Identify control and treatment groups
3.        Identify KPIs to measure
•        Main KPI
•        MOI : Measures of interest, , give more detail let us find out why reject or accept
4.        Identify what data needs to be collected
5.        Make sure the appropriate logging is in place to collect all necessary data
6.        Determine how small of a difference you would like to detect: tolerance of error
7.        Determine what fraction of visitors you want to be in the treatment: % of visitor in treatment group, how many traffic give treatment group
8.        Run power analysis to decide how much data you need to collect and how long you need to run the test
9.        Run the test for AT LEAST this long
10.        First time trying something new: run an A/A test simultaneously to check for systematic biases. All people in control group or all people in experience group

Tips: Only one thing should change between treatment and control .
Should have 3 groups:  2 difference: 2 vs 1 for bounce rate/ 3 vs 2 for purchase rate(transaction)
1. plain page
2. personalized home page
3. Personalized home page + content page