DS mock interview 分享－A/B testing, product case study, statistical learning

跟大家分享近期跟朋友 mock interview practice，笔记是英文写的所以直接就贴英文了，希望听一听大家的反馈，有不同想法的都可以回帖交流，互相学习。

本人背景：Applied Econ, Ph.D.，postdoc一年多，计划找DS工作，系统学习准备面试ing，如果有人愿意一起学习，一起 mock interview，也欢迎回帖联系。

＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝
Experiment Case: A new feature for search page  

Results from search query, recommender system is not very diverse, so the Ecommerce company thinks about a new feature to help user search more efficiently and diversely.
1. Define a metric to measure whether the change is good to business or not?
2. How do you test this new feature?

In the beginning, make sure you understand the question clearly, summarize the problem, then list the plan in a systematic format:

First, define a few metrics to be considered

• Sensitivity: a good metric should be correlated with user interaction, i.e., users need to use this feature
• Robustness: relevant to business goals, i.e. important to business growth or profits
• Not affected by other features, i.e. things you can't control should not contaminate your metric measurements
  

Think about the funnel analysis, visualize how a user interact with the product/the website, possible metrics: click-through rate, click-through probability, conversion rate, etc. CTP is more robust than CTR since CTR can be easily contaminated by other existing features. In this case, statistical analysis may not be reliable.

Explain how the selected metrics may impact business goals, ask the interviewer which one to go for.

Second, set up the experiment
1. Translate the business problem into statistical testing, i.e., A/B testing
2. Define your null hypothesis, select the test statistic and the underlying assumptions
     - Test two CTPs under null and under new feature
     - Sample size is large enough to apply CLT
     - Use Z-score test statistic
3. According to business goals, set up significance level, effect size, statistical power
     - Type 1 error vs type 2 error
     - False positive rate, false negative rate
     - Power analysis, tradeoff
     - Ask the interviewer which factor is more important to business
4. Calculate required minimum sample size and experiment duration
     - What is baseline conversion rate, what is expected effect size
. check 1point3acres for more.     - How to sample users, randomization according to traffic, blocks, or other separable features, like regions, locations, devices, depending on the features
     - What if can't do pure randomization, cluster randomization
     - How long the experiment should run, generally 2~4 weeks
     - What if I find a statistically significant result with minimum sample size, should I stop the experiment? NO, need consider user time to adopt, or user novelty effect, meaning that early observations may not be representative, start counting observations when sample data is relatively stable. Also, think about holidays, seasons, weekends, big events, try to cover broadly enough, not too short nor too long. Since there is a tradeoff. Longer time means that you may lose opportunity to launch new experiments.

Third, collect the results and make a decision about to launch or not
    - Back-of-the-envelope calculation, cost analysis
    - If launch, what is the loss if due to false positive rate, what is the expected cost for launch
    - If not launch, what is the expected gain from the significant results
    - Opportunity cost, tradeoff for other experiment, etc.

Fourth, back to the selected metrics and the business questions
    - Always tie your analysis to the business questions and business goals
    - Think about other metrics, how to improve, what not be considered, etc.


Statistical modeling
-baidu 1point3acres
1. Can you explain what is overfitting? And how do you detect overfitting?
   - Bias-variance tradeoff
   - High bias: underfitting (simpler model/hypothesis class)
   - High variance: overfitting (more complex model/hypothesis class)
   - Overfitting models have larger prediction errors than training errors
   - When doing model selections (or feature selections), we face a bias-variance tradeoff, we can apply cross validation to track and compare training vs testing errors as a guidance for model selection.
   - Give an example: linear regression vs polynomial of p-order, higher order polynomials tend to overfit new dataset  

2. Can you explain what is K-fold cross validation? How to determine the value of k?
   - k-fold cross validation is a method used to do model selections. Say we have multiple models and we want to know which one is better performed.
   - One way to do is to split the whole datasets to training, validation, and testing sets. Here k means how much subsets we want to split between training and validation sets. After splitting into K-fold subsets, we train each of our candidate models on the k-1 subsets and validate on the k-th subset, we then calculate the average errors for each candidate model and select the one with lowest average errors.
   - Regarding the value of k, if we have a large sample size, a common split is 80% vs 20% training-testing sets. However, if we have a smaller sample size, we may want to use leave-one-out cross validation, which allow us to have much more training data than testing data. In other words, we try to overfit the model given smaller sample size. Again, bias-variance tradeoff applies here.

3. For a binary outcome with many features, how do you assess the performance of many different classifiers?
   - Precision = accuracy of positive predictions = TP/(TP+FP)
   - Recall = sensitivity = 1 – type II error (β) = TP/(TP+FN)
   - False Positive Rate = 1 - specificity = type I error (α) = FP/(TN+FP)
   - False Discovery Rate = 1 - precision = FP/(TP+FP)
   - F1-score = harmony average of precision and recall, more preferred to use in unbalanced classification models

4. How to choose different classifiers before choosing the threshold of prob(Y=1)?
   - AUC, area under ROC, a plot of Recall (power, 1-β) vs False positive rate (α). Larger AUC, better model to choose.
   - Or choose the model that have lower AIC, BIC values.

5. Given the above two classifiers, which model would you choose if you require a sensitivity of at least 0.7? And if need the specificity to be at least 0.9?
   - Sensitivity = power >= 0.7, we want to choose the black ROC model.
   - Specificity = significance level >= 0.9, we want to choose the red ROC model.

5-b. You are a doctor who has built classifier(s) for cancer based on the results of blood tests. Which would you choose (red or black model) and what would you set the threshold in each of the following scenarios:
5-b1: You will administer chemotherapy to the patients classified as having cancer.
5-b2: You schedule more tests (more expensive than the first test but less than chemotherapy) to better evaluate whether a patient has cancer.
   - If we just do one test, we want to minimize False positive rate (have higher significance level), thus, want to choose red ROC model.
   - If we do multiple tests, in the first test we want to control sensitivity (give higher power), which gives us lower false negative (lower type II error). In the second test we then want to control False positive rate (lower type I error) for those classified as positive from the first test.
   - Thus, for multiple tests, we choose black ROC model in the first test, then red ROC model in the second test.

6. With plenty of features available, how should we select the most important features?
. .и   - Ask how big the sample size is, why model with features is not preferred, what algorithms/models will be used, what is the goal of the modeling, understand the context before answering.
   - If we have a small sample size, and we have P>>N, we can’t do a classic regression model because (XTX) is not full rank, not invertible, meaning that we can’t solve model coefficients.
   - There are a few methods to do feature selections. Forward selection, iteratively adding new feature that significantly improve model fit. Backward elimination, iteratively dropping existing feature that insignificantly decrease model fit. Here model fit metrics can be AIC, BIC, etc.
   - If we have large sample size, but we want to reduce overfitting. We can consider regularization methods such as LASSO, which will force coefficients to be zero for those insignificant features.
   - We can also use PCA to reduce feature dimensionality.
   - Another thing to consider is multicollinearity. Do some features have high covariance? With the presence of multicollinearity, estimated standard error of coefficient will be larger, leading to higher False negative error (higher type II error), or equivalently, lower power.
   - Do we care about feature interpretability (point estimate and confidence interval)? If we just care about prediction, multicollinearity is fine to be ignored since point estimates are not affected. But the confidence interval of point estimates will be wider and thus interpretation of coefficients may be invalid.
   - We can use Ridge regression to shrink the variance of estimated coefficients, at a tradeoff of introducing some bias. Ridge/LASSO regression increase the power of estimates.
   - Random forests algorithms are more robust to multicollinearity.

7. Can you explain what is decision tree, random forest, bagging, and boosting algorithms?
   - CART decision tree, keep splitting nodes based on a feature cutoff threshold.
   - Bagging, bootstrap aggregation, random sampling with replacement, train on these samples using decision trees and aggregate to get an average classifier.
   - Random forest, random sampling on a subset of features with replacement, train and aggregate to get an average classifier. Random forest introduces more bias, but gives less variance.
   - Boosting, random sampling but with more weights on misclassified observations, train and get a weighted average classifier. Boosting introduces more variance, but gives lower bias.


Product case study: new search ranking algorithm

In an E-commerce company, we design a new ranking algorithm for search query.
1. How do you measure whether this new ranking algorithm works or not? What metric(s) can be used to assess its performance?
2. What other factors/features do you think may affect its performance? How do you know? List these factors in buckets and present them in decreasing order of importance.

Think about more metrics, not just those often-used ones such as click-through-probability, conversion rate, etc.

Although I thought of a quality score to measure the performance of ranking algorithm, I didn't elaborate it in a good formula. My thought was to weighted average the CTPs over top k slots, with each slot weight being decreasing with respect to their corresponding position. Then we calculate and compare the quality score for both old and new ranking algorithms. We would prefer new ranking algorithm if the calculated quality score is significantly better than that from the old algorithm. My idea is somewhat similar to the DCG metric, though I didn’t know any of them before searching online.  

More metrics to evaluate search ranking algorithm:
   - Discounted cumulative gain (DCG) – a canonical metric that captures the intuition that the higher the result in the SERP (search engine result page), the more important/relevant it is to get it right.
   - Normalized discounted cumulative gain (NDCG)
   - Precision@k
   - Recall@K
   - Mean average precision
   - Mean reciprocal precision

Follow up question: What if the overall performance of new ranking algorithm is better than the old one, but the first slot CTP is lower than the old one? Do you decide to launch the new algorithm or not?

您好！
本帖隐藏的内容需要积分高于 150 才可浏览
您当前积分为 0。
使用VIP即刻解锁阅读权限或查看其他获取积分的方式

游客，您好！
本帖隐藏的内容需要积分高于 150 才可浏览
您当前积分为 0。
VIP即刻解锁阅读权限 或 查看其他获取积分的方式
Unlock interview details and practice with AI
Curated Interview Questions from Top Companies

. 1point3acres.com









补充内容 (2021-06-21 23:49 +8:00):
楼主借的朋友的账号发的帖，感谢大家的支持点赞，如果觉得有用，烦请多加大米大米哟~ 希望大家都能成功上岸，找到心仪的工作 ;)

wechat: nicole_0412

谢谢楼主分享很详细的笔记！我是数学Ph.D.，最近也在准备DS的面试内容，希望和楼主一起mock和学习！WeChat：liaoshasha0216

很详细的DS笔记，学习一下！

码住 感谢楼主分享！！

你好！可以联系我！15123943622

你好， 求加入。 我有几年DS工作经验， 在准备在职跳槽。 希望能加入一起加油分享经验~   lighterkey

感谢楼主的分享！

求加入 我也是Econ phd 背景 Wechat: gjy915203293

很详细的笔记，学习mark一下！

感谢楼主的分享！很详细的笔记～谢谢！

楼主借的朋友的账号发的帖，感谢大家的支持点赞，如果觉得有用多加大米大米哟~