楼主你好, 你写的这两个 AB TESTING 问题,想请教一下。
10). If after running an A/B testing you find the fact that the desired metric(i.e, Click Through Rate) is going up while another metric is decreasing(i.e., Clicks). How would you make a decision?
so we need to rethink the purpose of our optimization, to increase CTR or Clicks and choose one as our target metric.
Alternatively, we can include both metrics and come up with an overall evaluation criterion in which we assign weights to different metrics..1point3acres
11). Now assuming you have an A/B testing result reflecting your test result is kind of negative (i.e, p-value ~= 20%). How will you communicate with the product manager?
If given the above 20% p-value, the product manager still decides to launch this new feature, how would you claim your suggestions and alerts?
Assuming your alpha is set at 0.5, with a p-value of 20%, we can't conclude that the new design/feature is effective at increasing the metric of interest.
If he/she still decides to launch it, he will commit type i error... 这样答是不是太粗略了,求楼主指教,谢谢!