DS面试中A/B test部分notes整理（干货）

大米还是不够看面经(好多要188+)，附上最近整理的A/B test 部分知识点，求大米，大家一起加油，面试顺利offer多多^_^


. From 1point 3acres bbs    • not practical to run a test for every single idea, expensive, cost…
        ○ conduct quantitative analysis using historical data to obtain the opportunity sizing of each idea: like upper-bound sizing of the impact by user size
    • What is A/B test:
        ○ a.k.a controlled experiments
        ○ Widely used in industry to make product launch decisions.
        ○ It allows tech companies to evaluate a product/feature with a subset of users to infer how the product may be received by all users.
        ○ Control A with existing features; Treatment B with new features
        ○ Evaluating features with a subset of users
    • Design A/B test:
        ○ How long to run a test
            § Sample size
                • The rule of thumb is that sample size n approximately equals 16 (based on α = 0.05 and β = 0.8) multiplied by sample variance divided by δ square, whereas δ is the difference between treatment and control, This formula is from Trustworthy Online Controlled Experiments by Ron Kohavi, Diane Tang and Ya Xu.
                    ® we need more samples if the sample variance is larger, and we need fewer samples if the delta is larger.
                    ® Sample variance can be obtained from the existing data
                    ® δ, i.e the difference between treatment and control, use the minimum detectable effect, i.e. 0.1% increase in revenue as the minimum detectable effect
                           
                        ◊ Type II error rate β or Power, because Power = 1 — β. You know one of them, you know the other.
                        ◊ Significance level α
                        ◊ Minimum detectable effect
            § Sample size -> # of days to run the experiment = sample size /# of users in each group
            § If the number is less than a week, we should run the experiment for at least seven days to capture the weekly pattern. It is typically recommended to run it for two weeks. When it comes to collecting data for a test, more is almost always better than not enough. But if need longer time:
                • short develpment cycles, incurs engineering cost, non-persistent user tracking and interactions 等。 可以找proxies, model user learning, 找 surrogates。本质上就是把长期效果用其他相关变量来代替。
        ○ Interference between variants
            § Typical design: split users randomly, users are independent assumptions
            § Assumption fails:
                • Network effect, user behaviors impacted by others, effect spillover the control group, usually more treatment effect
                • Two-sided markets, resources are shared among control & treatment: actual effect < treatment effect
            § Solution:
. .и                • Isolate users between control and treatment:
                    ® By geolocations, i.e. NY & SF
                    ® By time
                    ® Create network clusters
                        ◊ Interact mostly within the clusters
                        ◊ Ego-network randomization: user either has the feature or not
                    ® Each has limitation, could combine

    • Analyzing Results
        ○ Multiple testing problem: multiple variants, so multiple testing running at same time
            § Do not use same significance level, probability of false discovery increases
            
            § Solution:
                • Conservative one: Bonferroni correction: significance level/# of tests = 0.05/10 = 0.005
                • Control False Discovery Rate (FDR)
               
        ○ Novelty & Primacy effect
            § Primacy effect (change aversion): People are reluctant to change
            § Novelty effect: People welcome the changes and use more
            § Both effects will not last long, so if A/B test has some initial effect, may due to novelty or primacy effect
            § Solution: run tests only on new users or compare new users to old users
        ○ treatment effect与实验当中的其他变量并不独立:
            § 不同X上效果截然相反的情况。虽然现实当中不一定相反，但已经直接影响了实验的可靠性
            § 可以对X的不同值做分别解析。例如根据 market/country, user activity level, device & platform, time and day of week, and product specific 划分。
            § computation scale, low SNR, multiple testing, interpretable and memorable等等问题。可以用on-demand or scheduled, sparse modeling, and merge segments等方法来解决。

    • Making Decisions
        ○ contradicting results
        ○ Although various factors are taken into consideration, such as the complexity of implementation, project management effort, customer support cost, maintenance cost, opportunity cost, etc.
        ○ But, focus current objective of the Question


Reference:
https://towardsdatascience.com/7 ... rviews-eee6428a8b63
https://www.optimizely.com/optim ... %20essentially%20an

终于可以看面经了， 非常感谢🙏

谢谢分享

棒棒 谢谢分享楼主