📣 黑五价提前开抢|VIP一年通行证5折,蓝莓、Offer多多同步特惠
查看: 20393| 回复: 25
跳转到指定楼层
上一主题 下一主题
收起左侧

udacity ab testing 打卡+总结

   
全局:
公开课
学校名称: google
Unit号: 1
开课时间: 2019-06-03
课程全名: A/B Testing
平台: Udacity
URL: https://www.udacity.com/course/ab-testing--ud257

注册一亩三分地论坛,查看更多干货!

您需要 登录 才可以下载或查看附件。没有帐号?注册账号

x
Lesson 1 Overview of A/B Testing


1-3. intro to a/b testing
  • useful: 1) help you climb to the peak of your current mountain
  • not useful: 1) test new experiences; old users may resist against the new version (change aversion), or old users may all go for the new experience, then the test set has everything (novelty effect). Two issues to consider when it comes to a new experience: (1) what is your baseline of comparison? (2) how much time do you need in order for your users to adapt to the new experience, so that you can actually say what is the plateaued experience and make a robust decision? 2) long term effect is hard to test too, such as a website that recommends apartment rentals; note update brand or logo can't use ab testing, it's surprisingly emotional and users need some time to get used to new logo 3) can't really tell you if you're missing something

1-9. metric choose

  • CTR: click-through-rate

     number of clicks   
number of page views
  • Click-through-probability: the number of unique visitors who click at least once divided by the number of unique visitors who view the page
unique visitors who click
unique visitors to page

1-19 Pooled Standard Error


1-20. practical, or substantive, significance
Statistical significance is about repeatability.
1-21 size V.S. power trade-off
Statistical power helps us find how many samples do we need to have statistical significance.
1-22 How Page Views Affect Sensitivity
推荐阅读:https://likan.info/cn/post/2017-08-11-p-values-bayes-factor/
1-beta = sensitivity = statistical power
1-25 calculation example


1-26 confidence interval case breakdown


2. neutral change
4. the confidence interval bounds are outside of what's practically significant, it could be causing them to increase or decrease
5. confidence interval overlaps 0, there might not even be a change at all
6. it's possible your change is not practically significant

评分

参与人数 16大米 +21 收起 理由
CherryMelon + 1 给你点个赞!
shiny23 + 1 很有用的信息!
Bgg + 1 good
Adam_Smith_ + 2 很有用的信息!
applephone + 1 给你点个赞!

查看全部评分


上一篇:[打卡]CS61B/61C
下一篇:Java零基础学习:Java Programming: Solving Problems with Software
推荐
 楼主| Briannabi 2019-6-8 10:39:51 | 只看该作者
全局:
Lesson 3 Choosing and Characterizing Metrics3-2 Metric Definition Overview
Types of Metrics:
1) Sanity checking metrics (Invariant Metrics) — can be multiple metrics. Eg. When changing the page and testing CTR, latency & load times should be invariant
2) Evaluation metrics — one or multiple.
For multiple metrics, composite metrics and OEC (overall evaluation criterion), they are hard to define.
3-3 Metric Definition Overview part 2
Steps for Metric Definition: 1) come up with a high-level concept for a metric, such as “active users”, which can be understood by everyone; 2) details: how do you define “active”; 3) summarize all of these individual data measurements into a single metric, such as count or sum;
3-5 Refining the Customer funnel
Udacity example:
  • Homepage visits
  • Exploring the site

       - Users who view course list
       - Users who view course details
  • Create an account
       - Users who enroll in a course
       - Users who finish lesson 1, 2, etc
       - Users who sign up for coaching at various levels
  • Completing a course

       - Users who enroll in a second class
       - Users who get jobs


3-6 Choosing Metrics


3-7 Difficult metrics
  • Don’t have access to data
  • Take too long to measure

3-8 Defining Metrics: Other Techniques
  • External data: Use industry metrics from External data (Neilson, comscore), Survey data (Pew, Forrester), Academic research.
  • Internal existing data: a. Retrospective analysis (correlation not causation), running experiments; b. user experience research, surveys, and focus groups

3-9 Techniques to Gather Additional Data
  • User Experience Research (UER)

    + Good for brainstorming
    + Can use special equipment
    - Want to validate results
  • Focus Groups

    + Get feedback on hypotheticals
    - Run the risk of group think
  • Surveys

    + Useful for metrics you cannot directly measure
    - Can’t directly compare to other results
3-13 Metric Definition
High-level metric: click-through-probability = users who click / users who visits
Def #1: for each time interval, cookies that click/cookies
Def #2: pageviews with click within time interval/pageviews
Def #3: clicks/pageviews (click-through-rate)
3-14 Filtering and Segment
Filter spam and fraud to de-bias your data
Compute your metric on a bunch of disjoint sets to check your disproportionately
3-16 Summary Metrics
Establish a few characteristics for your metric.
  • Sensitivity and robustness
  • Characterize the distribution: retrospective analysis

  • sums and counts
  • Distribution metrics: means, median and percentiles(75th, 90th)
  • probabilities and rates
  • Ratios

3-17 Sensitivity and Robustness
Sensitivity: a metric that picks up changes you care about
Robustness: is robust against changes that you don’t care about, it doesn’t move a lot when nothing interesting has happened.
How to measure them?
  • Running experiments, a/a experiment to determine if they’re too sensitive
  • Retrospective analysis of your logs

3-20 Absolute or Relative Difference (percent change)?
seasonality
3-21 Variability
变异,指标一般是标准差、方差、变异系数
3-22 Variability example
Calculating variability
To calculate a confidence interval you need:
  • Variance
  • Distribution

type of metricdistribution estimated variance
probabilitybinomial qrt(p*(1-p)/N)
meannormal sigma^2/N
count/differencenormal(maybe) var(x)+var(y)
median/percentiledepends depends
ratesPoisson mean(x)
ratiopoisson mean(x)



3-23 Nonparametric Answers
Sign test
3-24 Empirical Variability
For complicated metrics, the distribution can be weird, you might want to shift to an empirical estimate.
A/A testing: control A against another control A, and there’s actually no change in what the users are seeing. What that means that any differences that you measure are due to the underlying variability, maybe of your system, of the user population, what users are doing, all of those types of things.
Pros: if your experiment system is itself complicated, it’s actually a good test of your system
Bootstrap: randomly divvy the big sample into a bunch of small samples and you do comparison within those random subsets. In reality, what we have is a whole gradation of different methods. If the bootstrap estimate is agreeing with your analytical estimate, you can probably move on. If not, consider big A/A testing.

Uses of A/A tests:
  • Compare results to what you expect (sanity check)
  • Estimate variance and calculate confidence
  • Directly estimate confidence interval


Example: https://docs.google.com/spreadsheets/d/17wWNY2jkDlG9BDMYQq2l-ku_8HGajXuF2Zvy__dBEL4/edit#gid=0



回复

使用道具 举报

推荐
 楼主| Briannabi 2019-6-11 08:01:48 | 只看该作者
全局:
Lesson 4 Designing an Experiment
4-1 Introduction
  • Choose “subject”
  • Choose “population”
  • Size
  • Duration

4-2 Unit of Diversion Overview
A unit of diversion is how we define what an individual subject is in the experiment.
You are doing an event based diversion, a cookie-based, or a user id, these are all what we call our unit of diversion.
4-3 unit of diversion example
Commonly used:
  • User id (log in)
       - Stable and unchanging
       - Personally identifiable

  • Anonymous id (cookie)
       - Changes when you switch browser or device
       - Users can clear cookie
  • Event
       - No consistent experiment
       - This is only appropriate in situations where the changes are not user-visible

Less common:
  • Device id
       - Only available for mobile
       - Tied to specific device
       - Unchangeable by user
  • IP address
       - Changes when location changes

4-4 Consistency of diversion
Three main considerations when choosing diversion:
  • User consistency
       - Most users won’t notice ranking changes, so event-based diversion is good
       - event<cookie<id
  • Ethical consideration for diversion
  • Variability

4-8 Unit of Analysis vs. Unit of Diversion
Unit of analysis: the denominator of your metric
  • When your unit of analysis is the same as unit of diversion, such as the page view, then the variability tends to be lower and close to the analytical estimate.
  • When you’re doing event-based diversion every single event is a different random draw, and your independence assumption is actually valid. When you’re doing cookie or user ID based diversion, the independence assumption is no longer valid because u’re actually diverting groups of events, they’re actually correlated together. And that will increase your variability greatly.

4-10 Inter- vs. Intra-User Experiments
  • Intra-user Experiment: you expose the same user to this feature being on and off over time.
  • Interleaved experiment: expose the same user to the A and the B side at the same time. Typically, this only works in cases where you’re looking at reordering a list.
  • Inter-user experiment: you’ve got different people on the A side and on the B side.

4-12 Target population: example


The margin of error is smaller than the observed differences, so the confidence interval will not include 0, the result is significant.


In this case,  the variability of the global data as measured by the pooled standard error is lower than the filtered data. Mostly because there is so much more data globally.
m=SEpool*1.96=0.0025
Since the margin of error is wider than the estimated differences, the confidence interval will include 0, which means this difference is not significant.
4-13 population vs. cohort
Cohort: you define an entering class and you only look at users who entered your experiment on both sides around the same time.
Use cohort instead of population when you are looking for learning effects, examining user retention, want to increase user activity, or anything requiring user to be established.
4-16 Sizing
How to reduce the size of an experiment to get it done faster? You can increase the practical significance boundary, significance level alpha, or reduce power (1-beta) which means increase beta, or change the unit of diversion if originally it is not the same with unit of analysis. Or you
can target experiment to specific traffic, unrelated traffic will dilute the result and impact choice of practical significance boundary.
4-20 Duration vs. Exposure
What fraction of your traffic you’re going to send through the experiment? The duration of your experiment is related to the proportion of traffic you’re sending through your experiment.
We couldn't run on all of traffic to get results quicker for several reasons, 1) safety to make sure the function works well; 2) press, we don't want a lot of people to know it before it works; 3) other things impact the variability, such as holiday
4-23 Learning Effect
  • When there’s a new change, in the beginning, users may against the change (change version) or use the change a lot (novelty effect). But over time, user behavior becomes stable, which is called the plateau stage. The key thing to measure learning effect is time, but in reality, you don’t have that much luxury of taking that much time to make a decision. Suggestion: run on a smaller group of users, for a longer period of time.
  • If you do have much time, you also need to notice the following points: 1) choosing the unit of diversion correctly; 2) dosage: how often we see the change, then you probably want to be using cohort as opposed to just a population; 3) risk and duration
  • Pre-periods and post-periods: A/A test before and after the experiment.







回复

使用道具 举报

🔗
 楼主| Briannabi 2019-6-6 04:36:54 | 只看该作者
全局:
June 5
Lesson 2 Policy and Ethics for Experiments
2.1 Tuskegee and Milgram Experiments
We need to ensure the participants are adequately protected. In both cases, participants were subject to a high risk of permanent harm.
2.2 Facebook Experiment
Four main principles:
1) Risk: what risk is a participant going to being exposed to?
2) Benefit: what benefits might be the outcome of the study?
3) Choice: what other choices do participants have?
4) Privacy: what expectation of privacy and confidentiality do participants have?
2-7 Data Sensitivity
Data that is aggregated to a granularity that does not allow re-identifying individuals is usually not considered sensitive.
2-9 Questions and Consent
1) Are users being informed?
2) What user identifiers are tied to data?
3) What type of data is being collected?
4) What is the level of confidentiality and security?
回复

使用道具 举报

🔗
 楼主| Briannabi 2019-6-12 07:37:49 | 只看该作者
全局:
Lesson 5 Analyzing Results
Sanity check
  • Choosing invariant metrics

    - Two types: population sizing metrics; any other metrics you don’t expect to change
  • Checking invariants

Example: experiment for two weeks
Total control:64454 Total experiment: 61818
Whether the difference is within expectations?
Solution 1:
1. Compute standard deviation of binomial with probability 0.5 of success
SD=sqrt(0.5*0.5/(64454+61818))=0.0014
2. Multiply by z-score yo get margin of error
m=SD*1.96=0.0027
3. Compute confidence interval around 0.5
0.4973 to 0.5027
4. Check whether observed fraction is within interval
p=64454/(64454+61818) =0.5104 which is greater than 0.5027, so something is incorrect.
        Solution 2:
        Checking whether any particular day stands out as causing the problem or whether it seems to be an overall pattern.

Single metric
The confidence interval does not include the practical significance boundary of 0.01, meaning I can be confident at a 95% confidence level that the true change is large enough to be worth launching.
Sign test
Online calculator: https://www.graphpad.com/quickcalcs/binomial1.cfm

The two-tail p value is 0.0158, which is less than 0.05.
Simpson's Paradox
Within each subgroup, the results are stable but when you aggregate them all together, it’s the mix of subgroups that actually drives your result.
https://zhuanlan.zhihu.com/p/21068622

Multiple metrics As you test more metrics, it becomes more likely that one of them will show a statistically significant result by chance.
For 3 metrics, what’s the chance of at least 1 false positive?
P(FP=0)=0.95^3=0.857 Assuming independence
P(FP>=1)=1-0.857=0.143
Problem: probability of any false positive increases as you increase number of metrics
Solution: Use higher confidence level for each metric
Method 1: Assume independence
Alpha_overall = 1-(1-Alpha_individual)^n
Method 2: Bonferroni Correction
  • no assumption
  • Conservative: guaranteed to give Alpha_overall at least as small as specified

Other methods:
  • FWER(familywise error rate): control probability that any metric shows a False Positive
  • FDR (false discovery rate): across a large number of metrics

FDR = E[false positives/rejections]


Drawing Conclusions
  • Do I have statistically significant and practically significant results in order to justify the change?
  • Do I understand what that change has actually done with regards to user experience?
  • Is it worth it?


Gotchas: changes over time
When you are ramping up the change, you may see the effect flatten out. Thus making the tested effect not repeatable. There are many reasons for this phenomenon.
  • Seasonality effect, event-driven impact; Solution: use hold-back method, launch the change to everyone except for one small hold-back group of users, and continue comparing their behavior to the control group.
  • Novelty effect or change aversion: cohort analysis may be helpful.


Lessons learned
  • Double check, triple check that your experiment is set up properly.
  • Not only think about statistically significant but also business impact. Think about the engineering cost, customer support or sales issue, what’s the opportunity cost, etc.
  • If you are running your first experiment that has a big impact, you might want to run a couple of experiments and check the results to see if you are comfortable launching it.
  • It is actually a business decision rather than a problem set of statistics.



回复

使用道具 举报

🔗
 楼主| Briannabi 2019-6-13 07:50:09 | 只看该作者
全局:
final project
最后的final project 参考
回复

使用道具 举报

🔗
HHHHarold 2019-11-11 12:01:02 | 只看该作者
全局:
感谢楼主,这几天正好在看
回复

使用道具 举报

🔗
qqsymsf 2020-1-3 04:37:27 | 只看该作者
全局:
感谢楼主总结!
回复

使用道具 举报

🔗
qqsymsf 2020-1-7 01:10:15 | 只看该作者
全局:
感谢楼主总结!
回复

使用道具 举报

🔗
zoeyrz 2020-5-10 23:58:17 | 只看该作者
全局:
感谢楼主分享!!
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 注册账号
隐私提醒:
  • ☑ 禁止发布广告,拉群,贴个人联系方式:找人请去🔗同学同事飞友,拉群请去🔗拉群结伴,广告请去🔗跳蚤市场,和 🔗租房广告|找室友
  • ☑ 论坛内容在发帖 30 分钟内可以编辑,过后则不能删帖。为防止被骚扰甚至人肉,不要公开留微信等联系方式,如有需求请以论坛私信方式发送。
  • ☑ 干货版块可免费使用 🔗超级匿名:面经(美国面经、中国面经、数科面经、PM面经),抖包袱(美国、中国)和录取汇报、定位选校版
  • ☑ 查阅全站 🔗各种匿名方法

本版积分规则

>
快速回复 返回顶部 返回列表