中级农民
- 积分
- 103
- 大米
- 颗
- 鳄梨
- 个
- 水井
- 尺
- 蓝莓
- 颗
- 萝卜
- 根
- 小米
- 粒
- 学分
- 个
- 注册时间
- 2019-3-7
- 最后登录
- 1970-1-1
|
Lesson 4 Designing an Experiment
4-1 Introduction
- Choose “subject”
- Choose “population”
- Size
- Duration
4-2 Unit of Diversion Overview
A unit of diversion is how we define what an individual subject is in the experiment.
You are doing an event based diversion, a cookie-based, or a user id, these are all what we call our unit of diversion.
4-3 unit of diversion example
Commonly used:
- User id (log in)
- Stable and unchanging
- Personally identifiable
- Anonymous id (cookie)
- Changes when you switch browser or device
- Users can clear cookie
- Event
- No consistent experiment
- This is only appropriate in situations where the changes are not user-visible
Less common:
- Device id
- Only available for mobile
- Tied to specific device
- Unchangeable by user
- IP address
- Changes when location changes
4-4 Consistency of diversion
Three main considerations when choosing diversion:
- User consistency
- Most users won’t notice ranking changes, so event-based diversion is good
- event<cookie<id
- Ethical consideration for diversion
- Variability
4-8 Unit of Analysis vs. Unit of Diversion
Unit of analysis: the denominator of your metric
- When your unit of analysis is the same as unit of diversion, such as the page view, then the variability tends to be lower and close to the analytical estimate.
- When you’re doing event-based diversion every single event is a different random draw, and your independence assumption is actually valid. When you’re doing cookie or user ID based diversion, the independence assumption is no longer valid because u’re actually diverting groups of events, they’re actually correlated together. And that will increase your variability greatly.
4-10 Inter- vs. Intra-User Experiments
- Intra-user Experiment: you expose the same user to this feature being on and off over time.
- Interleaved experiment: expose the same user to the A and the B side at the same time. Typically, this only works in cases where you’re looking at reordering a list.
- Inter-user experiment: you’ve got different people on the A side and on the B side.
4-12 Target population: example
The margin of error is smaller than the observed differences, so the confidence interval will not include 0, the result is significant.
In this case, the variability of the global data as measured by the pooled standard error is lower than the filtered data. Mostly because there is so much more data globally.
m=SEpool*1.96=0.0025
Since the margin of error is wider than the estimated differences, the confidence interval will include 0, which means this difference is not significant.
4-13 population vs. cohort
Cohort: you define an entering class and you only look at users who entered your experiment on both sides around the same time.
Use cohort instead of population when you are looking for learning effects, examining user retention, want to increase user activity, or anything requiring user to be established.
4-16 Sizing
How to reduce the size of an experiment to get it done faster? You can increase the practical significance boundary, significance level alpha, or reduce power (1-beta) which means increase beta, or change the unit of diversion if originally it is not the same with unit of analysis. Or you
can target experiment to specific traffic, unrelated traffic will dilute the result and impact choice of practical significance boundary.
4-20 Duration vs. Exposure
What fraction of your traffic you’re going to send through the experiment? The duration of your experiment is related to the proportion of traffic you’re sending through your experiment.
We couldn't run on all of traffic to get results quicker for several reasons, 1) safety to make sure the function works well; 2) press, we don't want a lot of people to know it before it works; 3) other things impact the variability, such as holiday
4-23 Learning Effect
- When there’s a new change, in the beginning, users may against the change (change version) or use the change a lot (novelty effect). But over time, user behavior becomes stable, which is called the plateau stage. The key thing to measure learning effect is time, but in reality, you don’t have that much luxury of taking that much time to make a decision. Suggestion: run on a smaller group of users, for a longer period of time.
- If you do have much time, you also need to notice the following points: 1) choosing the unit of diversion correctly; 2) dosage: how often we see the change, then you probably want to be using cohort as opposed to just a population; 3) risk and duration
- Pre-periods and post-periods: A/A test before and after the experiment.
|
|