谷歌 DS Onsite 过经

yusiyunteresa 发表于 2019-5-23 01:47
还要请问下，点击量服从power law的分布，分布的参数估计又和rank具体排第几 是怎么联系的？

I do not really know if this is the answer, but I would assume the probability of a video being viewed experience exponential decay with time, or the number of viewers experience exponential decay with time. Random variables with such characteristics are the exponential distribution, Gamma distribution, Prateo distribution, etc. A first guess would be a log transformation. If we are doing regression with an ordinal variable, it seems best to work with a Bayesian hierarchy model or even a non-parametric model in which the rank can be naturally introduced as a scoring feature. Alternatively we can fit a parametric model; since viewers may view the same video repeatedly, we can fit a random-effect model to control the potential clustering among the users. Judging from the information given by OP, the MLE should be asymptotically efficient assuming we are modelling the true model. If not the inefficiency can be measured by K-L divergence.

Note that since sample size is assumed to be large, a more robust estimator (versus the MLE) may perform better with some loss of efficiency.  

那个，如果分布服从power law，是不是就是一个y=ax^b然后我们想要estimate的是a和b，然后就是做个log transform，然后就是一个linear regression，然后随便用个ols或者mle结果是一样的，所以是asymptotically optimal的。。。

liu1p3 发表于 2019-5-25 10:32
那个，如果分布服从power law，是不是就是一个y=ax^b然后我们想要estimate的是a和b，然后就是做个log trans ...

I think the OLS estimator and MLE estimator can be different in general. You need the MLE and you should mention things like invariance property of MLE if you are consider asymptotic effiency in terms of the transformed variable. For a rigorous derivation, see here:

https://en.wikipedia.org/wiki/Pareto_distribution

Note this is very similar to another Google interview question; the estimation of a truncated normal distribution.

铅笔 发表于 2019/05/25 11:38:35


I think the OLS estimator and MLE estimator can be different in general. You need the MLE and you ...

嗯嗯，但是linear regression就是一样啊

liu1p3 发表于 2019-5-25 12:27
嗯嗯，但是linear regression就是一样啊

The fact MLE coincide with OLS estimator is a special case, not the general case. If you assume the variable satisfies the log-normal distribution, then after taking logarithm you may use the OLS estimator. Otherwise it makes little sense.

铅笔 发表于 2019/05/25 12:30:52



The fact MLE coincide with OLS estimator is a special case, not the general case. If you assume ...

也对，那不管怎么着我就只用mle好了…那个商场的买东西的题，你觉得是zip不？你刚才提到的那个truncated normal是用mle估计三个parameter对吗？

liu1p3 发表于 2019-5-25 12:43
也对，那不管怎么着我就只用mle好了…那个商场的买东西的题，你觉得是zip不？你刚才提到的那个truncated  ...

I am not following your comment. They asked to deal with missing value situation in Poisson regression. Depending on the specific missing scenario, it can be rather involved. For MCAR setting, I would consider fill in the data with a Gamma prior and using Gibbs sampling for imputation. I am not sure what "Zip" refers to at here. I think using Bayesian methods may be entirely overkill. Maybe there are better ways under the GLM framework using Fisher scoring or something.

铅笔 发表于 2019/05/25 13:12:17


I am not following your comment. They asked to deal with missing value situation in Poisson regres...

zero inflated poisson…

liu1p3 发表于 2019-5-25 13:23
. check 1point3acres for more.zero inflated poisson…

I think it is a good idea but sounds grossly simplified. Maybe I am wrong.

铅笔 发表于 2019/05/25 13:33:11


I think it is a good idea but sounds grossly simplified. Maybe I am wrong.

我决定怎么简单怎么来…我上次面别家面到missing，然后面试官说at random假设不对，我说那用pattern mixture啊，他说他不会…所以我学乖了