查看: 1774|回复: 10
收起左侧

哪些模型需要calibration,为什么?

|只看干货
chaofan | 显示全部楼层 |阅读模式
本楼: 👍   100% (3)
 
 
0% (0)   👎
全局: 👍   92% (92)
 
 
7% (7)    👎

注册一亩三分地论坛,查看更多干货!

您需要 登录 才可以下载或查看附件。没有帐号?注册账号

x
如题,最近在看model calibration的内容,发现有的需要calibration,如SVM, boosted trees,有的已经well calibrated,如logistic regression,NN。
原文及链接如下:

It is particularly effective for max-margin methods such as SVMs and boosted trees, which show sigmoidal distortions in their predicted probabilities, but has less of an effect with well-calibrated models such as logistic regression, multilayer perceptrons, and random forests.

https://kingsubham27.medium.com/ ... arning-71bec997b661

想请问一下为什么LR, NN, RF这些模型已经well calibrated了呢?求讨论解答

上一篇:MLE System Design Mock 组队 Machine Learning
下一篇:从日常生活理解variance-bias trade-off,真的很牛逼
Adam1679 2022-7-26 06:58:51 | 显示全部楼层
本楼: 👍   100% (1)
 
 
0% (0)   👎
全局: 👍   98% (79)
 
 
1% (1)    👎
Adam1679 发表于 2022-7-25 15:58
模型校准是在广告推荐模型比较老生常谈的问题,本质就是好的校准模型本后有个合理概率分布,比如逻辑回归背 ...

觉得有用求+1

评分

参与人数 1大米 +1 收起 理由
table42please + 1 赞一个

查看全部评分

回复

使用道具 举报

arrowcy 2022-7-23 15:03:14 | 显示全部楼层
本楼: 👍   100% (1)
 
 
0% (0)   👎
全局: 👍   100% (14)
 
 
0% (0)    👎
本帖最后由 arrowcy 于 2022-7-23 00:07 编辑

I am also trying to learn about this topic. Here is my current understanding:
A model is well calibrated if it uses log-loss with no regularization, uniform sample weights, no down-sampling / up-sampling

Logistic regression is well calibrated. I think this can be proved theoretically. But intuitively, for a number of observations with similar X, they should have similar prob output. So we can think the model will produce the same output prob, it should equal to the #pos / #total samples. Otherwise, the log-loss will be higher. That is because when using log-loss, the loss is minimized when the output prob has the same distribution as the observed distribution. Certainly, there may be some quantization error (because the y is binary but the model output is a real number in [0,1] ). So in this sense, as long as we use log-loss, the prob should be well calibrated.

NN may not be. Particularly, the widely used stochastic gradient descent *might* have some regularization effect and may impact the calibration of the output.
Random forest may not be. For example, the node split operation may not directly optimize the log-loss. So even if the leaf node used log-loss, the splitting condition has already biased the model.

Finally, if the model's capacity is not sufficient, the model may not be well-calibrated.
advertisement
回复

使用道具 举报

本楼: 👍   0% (0)
 
 
0% (0)   👎
全局: 👍   93% (1313)
 
 
6% (96)    👎
你有疑问就对了。因为nn 也需要calibrate。理论上任何model都要calibrate。
回复

使用道具 举报

本楼: 👍   0% (0)
 
 
0% (0)   👎
全局: 👍   100% (340)
 
 
0% (0)    👎
蹲一个回答。也在了解
回复

使用道具 举报

 楼主| chaofan 2022-7-24 07:58:39 | 显示全部楼层
本楼: 👍   0% (0)
 
 
0% (0)   👎
全局: 👍   92% (92)
 
 
7% (7)    👎
arrowcy 发表于 2022-7-23 00:03
I am also trying to learn about this topic. Here is my current understanding:
A model is well calib ...

Can you further explain why the gradient descent in NN will bias the model even if it uses log loss? Why it differs from LR? In my opinion, if #pos / #total samples is not equal in NN, the loss will also get higher, is that right?
回复

使用道具 举报

 楼主| chaofan 2022-7-24 08:00:54 | 显示全部楼层
本楼: 👍   0% (0)
 
 
0% (0)   👎
全局: 👍   92% (92)
 
 
7% (7)    👎
MichaelLee123 发表于 2022-7-22 18:47
你有疑问就对了。因为nn 也需要calibrate。理论上任何model都要calibrate。

那请问是不是nn相对已经比较well calibrated呢? 怎么分析一个模型是否well calibrated呢?
回复

使用道具 举报

arrowcy 2022-7-24 15:20:33 | 显示全部楼层
本楼: 👍   0% (0)
 
 
0% (0)   👎
全局: 👍   100% (14)
 
 
0% (0)    👎
本帖最后由 arrowcy 于 2022-7-24 00:27 编辑
chaofan 发表于 2022-7-23 16:58
Can you further explain why the gradient descent in NN will bias the model even if it uses log los ...

I found a few references about the implicit regularization effect of gradient descent. But those papers are quite difficult to read and comprehend for me. So I am not very confident if I am interpreting it in the right way.

You could start from this one:

Gradient descent follows the regularization path for general losses (https://arxiv.org/pdf/2006.11226.pdf)

It basically said that:
* if we run the gradient descent for many interation, we will get a path in the parameter space.
* if we run regularized regression, but relax the regularization strength towards 0, we will get a regularization path.
The conclusion is, the gradient descent path and the regularization path will get closer as gradient descent iterates, and as we relax the regulartion.

So maybe the gradient descent result (assuming in practice, we did not run for infinite number of iterations and the dataset is not big enough to ensure finding a true optimal) will be close to some slightly regularized regression result. In this sense, the gradient descent + log-loss may not be well calibrated.

Caution: the above paper is a purely theoretical and asymptotical study, so we cannot say for sure how big the bias is in practice. It's better to use calibration curve etc to check the calibration after training a model.
Why LR is different from NN:  I am sorry :( but the quick answer is that I am not sure. Because the paper's conclusion seems to be applicable to LR also. As I mentioned I am also learning about this topic recently.
回复

使用道具 举报

 楼主| chaofan 2022-7-26 06:43:21 | 显示全部楼层
本楼: 👍   0% (0)
 
 
0% (0)   👎
全局: 👍   92% (92)
 
 
7% (7)    👎
arrowcy 发表于 2022-7-24 00:20
I found a few references about the implicit regularization effect of gradient descent. But those p ...

Thank you for your explanation.
回复

使用道具 举报

Adam1679 2022-7-26 06:58:33 | 显示全部楼层
本楼: 👍   0% (0)
 
 
0% (0)   👎
全局: 👍   98% (79)
 
 
1% (1)    👎
模型校准是在广告推荐模型比较老生常谈的问题,本质就是好的校准模型本后有个合理概率分布,比如逻辑回归背后的假设是logistic distribution,nn的话主要是看loss怎么设计,常见的crossentropy,MSE都是有先验分布假设的,不需要校准。细节见 https://zhuanlan.zhihu.com/p/143099734

评分

参与人数 2大米 +2 收起 理由
jackzhou + 1 给你点个赞!
chaofan + 1 很有用的信息!

查看全部评分

回复

使用道具 举报

您需要登录后才可以回帖 登录 | 注册账号
隐私提醒:
  • ☑ 禁止发布广告,拉群,贴个人联系方式:找人请去🔗同学同事飞友,拉群请去🔗拉群结伴,广告请去🔗跳蚤市场,和 🔗租房广告|找室友
  • ☑ 论坛内容在发帖 30 分钟内可以编辑,过后则不能删帖。为防止被骚扰甚至人肉,不要公开留微信等联系方式,如有需求请以论坛私信方式发送。
  • ☑ 干货版块可免费使用 🔗超级匿名:面经(美国面经、中国面经、数科面经、PM面经),抖包袱(美国、中国)和录取汇报、定位选校版
  • ☑ 查阅全站 🔗各种匿名方法

本版积分规则

>
快速回复 返回顶部 返回列表