哪些模型需要calibration，为什么？

如题，最近在看model calibration的内容，发现有的需要calibration,如SVM, boosted trees，有的已经well calibrated,如logistic regression，NN。
原文及链接如下：

It is particularly effective for max-margin methods such as SVMs and boosted trees, which show sigmoidal distortions in their predicted probabilities, but has less of an effect with well-calibrated models such as logistic regression, multilayer perceptrons, and random forests.

https://kingsubham27.medium.com/ ... arning-71bec997b661

想请问一下为什么LR, NN, RF这些模型已经well calibrated了呢？求讨论解答

Adam1679 发表于 2022-7-25 15:58
模型校准是在广告推荐模型比较老生常谈的问题，本质就是好的校准模型本后有个合理概率分布，比如逻辑回归背 ...

觉得有用求+1

I am also trying to learn about this topic. Here is my current understanding:
A model is well calibrated if it uses log-loss with no regularization, uniform sample weights, no down-sampling / up-sampling

Logistic regression is well calibrated. I think this can be proved theoretically. But intuitively, for a number of observations with similar X, they should have similar prob output. So we can think the model will produce the same output prob, it should equal to the #pos / #total samples. Otherwise, the log-loss will be higher. That is because when using log-loss, the loss is minimized when the output prob has the same distribution as the observed distribution. Certainly, there may be some quantization error (because the y is binary but the model output is a real number in [0,1] ). So in this sense, as long as we use log-loss, the prob should be well calibrated.

NN may not be. Particularly, the widely used stochastic gradient descent *might* have some regularization effect and may impact the calibration of the output.
Random forest may not be. For example, the node split operation may not directly optimize the log-loss. So even if the leaf node used log-loss, the splitting condition has already biased the model.

Finally, if the model's capacity is not sufficient, the model may not be well-calibrated.

chaofan 发表于 2022-7-23 16:58
Can you further explain why the gradient descent in NN will bias the model even if it uses log los ...

I found a few references about the implicit regularization effect of gradient descent. But those papers are quite difficult to read and comprehend for me. So I am not very confident if I am interpreting it in the right way.

You could start from this one:

Gradient descent follows the regularization path for general losses (https://arxiv.org/pdf/2006.11226.pdf)

It basically said that:
* if we run the gradient descent for many interation, we will get a path in the parameter space.
* if we run regularized regression, but relax the regularization strength towards 0, we will get a regularization path.
The conclusion is, the gradient descent path and the regularization path will get closer as gradient descent iterates, and as we relax the regulartion.

So maybe the gradient descent result (assuming in practice, we did not run for infinite number of iterations and the dataset is not big enough to ensure finding a true optimal) will be close to some slightly regularized regression result. In this sense, the gradient descent + log-loss may not be well calibrated.

Caution: the above paper is a purely theoretical and asymptotical study, so we cannot say for sure how big the bias is in practice. It's better to use calibration curve etc to check the calibration after training a model.
Why LR is different from NN:  I am sorry :( but the quick answer is that I am not sure. Because the paper's conclusion seems to be applicable to LR also. As I mentioned I am also learning about this topic recently.

你有疑问就对了。因为nn 也需要calibrate。理论上任何model都要calibrate。

蹲一个回答。也在了解

arrowcy 发表于 2022-7-23 00:03
I am also trying to learn about this topic. Here is my current understanding:
A model is well calib ...

Can you further explain why the gradient descent in NN will bias the model even if it uses log loss? Why it differs from LR? In my opinion, if #pos / #total samples is not equal in NN, the loss will also get higher, is that right?

MichaelLee123 发表于 2022-7-22 18:47
你有疑问就对了。因为nn 也需要calibrate。理论上任何model都要calibrate。

那请问是不是nn相对已经比较well calibrated呢？ 怎么分析一个模型是否well calibrated呢？

arrowcy 发表于 2022-7-24 00:20
I found a few references about the implicit regularization effect of gradient descent. But those p ...

Thank you for your explanation.

模型校准是在广告推荐模型比较老生常谈的问题，本质就是好的校准模型本后有个合理概率分布，比如逻辑回归背后的假设是logistic distribution，nn的话主要是看loss怎么设计，常见的crossentropy，MSE都是有先验分布假设的，不需要校准。细节见 https://zhuanlan.zhihu.com/p/143099734