在职DA转DS找工/学习记录

本人BG,  一直在mkt行业做analysis，5+工作经验，想在今年转DS。

这周开始学习统计知识，笔记分享～和地里找工的小伙伴共享，会持续更新～

Selection bias: refers to the practice of selectively choosing data-consciously or unconsciously-in a way that leads to a conclusion that is misleading or ephemeral.
- selection bias: bias resulting from the way in which observations are selected
- data snooping: extensive hunting through data in search of something interesting
- vast search effect: bias or non-reproducibility resulting from repeated data modeling, or modeling data with large numbers of predictor variables
regression to the mean: Regression to the mean is all about how data evens out. It basically states that if a variable is extreme the first time you measure it, it will be closer to the average the next time you measure it. In technical terms, it describes how a random variable that is outside the norm eventually tends to return to the norm.

data distribution: the frequency distribution of individual values in a data set.
sampling distribution: the frequency distribution of a sample statistic over many samples or resamples.
central limit theorem: the tendency of the sampling distribution to talk on a normal shape as sample size rises.
standard error: the variability of a sample statistic over many samples. (Vs. standard deviation, refers to variability of individual data values).
standard error =SE= s/√n.  (s is the standard deviation of the sample values, n is the sample size)

bootstrap: a sample taken with replacement from an observed data set.
resampling: the process of taking repeated samples from observed data, includes both bootstrap and permutation(shuffling) procedures

The bootstrap doesn't compensate for a small sample size, it doesn't create new data, nor does it fill in holes in an existing data set, it is a powerful tool for assessing the variability of a sample statistics.


Confidence interval is a tool that can be used to get an idea of how variable a sample result might be.
The percentage associated with the confidence interval is termed the level of confidence. The higher the level of confidence, the wider the interval. Also, the smaller the sample, the wider the interval. The more confident you want to be, and the less data you have, the wider you must make the confidence interval to be sufficiently assured of capturing the true value.

Normal distribution: 68% of the data lies within one standard deviation of the mean, 95% lies within two standard deviations (Gaussian distribution)
error: the difference between a data point and a predicted or average value.
standardize: subtract the mean and divide by the standard deviation
z-score: the result of standardizing an individual data point
standard normal: a normal distribution with mean =0 and standard deviation =1
QQ-Plot: a plot to visualize how close a sample distribution is to a specified distribution.

t-distribution is a normally shaped distribution, except that it is a bit thicker and longer on the tails, which used to depict distributions of sample statistics.
Degrees of freedom, a parameter that allows the t-distribution to adjust to different sample sizes, statistics, and numbers of groups

binomial distribution is the frequency distribution of the number of successes in a given number of trails with specified probability of success in each trial.
with large n, and provided p is not too close to 0 or 1, the binomial distribution can be approximated by the normal distribution.

chi-square distribution, difference between the observed and expected values, divided by the square root of the expected value, squared, then summed across all categories. The chi-square distribution is typically concerned with counts of subjects or items falling into categories, it measures the extent of departure from what you would expect in a null model.

F-distribution, ANOVA is used with experiments and linear models involving measured data, compares variation due to factors of interest to overall variation.  

Poisson and related distribution: the frequency distribution of the number of events in sampled units of time or space
Exponential distribution: the frequency distribution of the time or distance from one event to the next event
Weibull distribution: a generalized version of the exponential distribution in which the event rate is allowed to shift over time.


A/B test, is an experiment with two groups to establish which of two treatments, products, procedures, or the like is superior.
Treatment, something to which a subject is exposed.
Treatment group, a group of subjects exposed to a specific treatment.
control group, a group of subjects exposed to no treatment
randomization, the process of randomly assigning subjects to treatments
subjects, the items that are exposed to treatments
test statistic, the metric used to measure the effect of the treatment

without control group, there is no assurance that "all other thins are equal" and that any difference is really due to the treatment.

Hypothesis Tests
null hypothesis, the hypothesis that chance is to blame.
alternative hypothesis, counterpoint to the null (what you hope to prove)
one way test, hypothesis test that counts chance results only in one direction.
two way test, hypothesis test that counts chance results in two directions.

we want a hypothesis to protect us from being fooled by chance in the direction favoring B, we don't care about being fooled by chance in the other direction, because we would be sticking with A unless B proves definitively better, we use one-way test

Statistical significance and p-Values
statistical significance is how statisticians measure whether an experiment yields a result more extreme than what chance might produce, if the result is beyond the realm of chance variation, it is said to be statistically significant.

请问lz有没有自学/巩固statistics的课程或者方法推荐？目前也在准备转ds，祝顺利！