- Deng, A., & Shi, X. (2016, August). Data-driven metric development for online controlled experiments: Seven lessons learned. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 77-86).
- Dmitriev, P., & Wu, X. (2016, October). Measuring metrics. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management (pp. 429-437).
- Chen, A. C., & Fu, X. (2017, April). Data+ intuition: A hybrid approach to developing product north star metrics. In Proceedings of the 26th International Conference on World Wide Web Companion (pp. 617-625).
. check 1point3acres for more.
3. 【google udacity abtesting的课程】. From 1point 3acres bbs
这个课我陆陆续续看过三次,心路历程如下。第一次:就这?原来ab testing就是个randomized clinical trials,so easy。。simpson’s paradox 学过学过,过。。第二次:说的还是挺有道理的。第三次(onsite前抱佛脚):原来人家可以把Simpsons paradox用这么几句话,说的这么清楚,学习了学习了。
5. 【多多思考】在回答product问题的时候,往往想到一些metric并不难。难的是继续追问自己:
- 这些metric大概的distribution是什么?is it normally distributed? what is the estimated mean, median, P90?
- 如果你想到了很多metrics,那么哪两三个是比较重要的?
- pros and cons of using this metric? is it sensitive to outliers? is it possible to capture bots?
- is this metric sensitive and robust? (参考udacity abtesting课). 1point3acres
- any edge cases? ..
- if this metric goes down, does it tell you something?
- if this metric goes up, but DAU goes down, does it tell you something?
6.【mock interviews】真的挺有帮助的,如果遇到不错的同学,你也可以从他们身上学习到一些;如果遇到不太好的,你也可以知道自己应该避免什么。
.google и
7. 【书籍】我买了两本cracking the PM interview and trustworthy online controlled experiments. 久仰diane tang and ya xu大名,就买了第二本书。我大概花了一天的时间,过了一下trustworthy这本书,挺有帮助的,比如里面有关怎么分析confidence intervals 的部分(page 37)。