分享一个没答好的consumer analytics题, missing value+imbalanced data+feature e...

最近交了一个coding题，feedback很不好. 感觉data 量太大，思路有点乱。题目如下， data和data dictionary放在我的google drive上

You are in a meeting with a product owner, they just launched an additional customer care center of excellence (CoE), and have opportunity to cross-sell/upsell customers with additional products. They need your help in identifying which customers to pitch to and what product.
At your disposal is historical data for the last 12 months of customer upgrades, their demographic and transaction data. The business has 4 products - video, data, home security and Voice. A customer might be subscribed to a single product, a combination of two or three products or have all 4 products.  
a). Describe how you would go about providing a solution to the business? What modeling approaches would you use? How would you measure the effectiveness of your solutions?
b). Now assume, the business partner has prepared a pipe delimited dataset for you. The dataset contains 128 variables and 300,000 records. The dataset contains records of customer that were serviced by the CoE in the last 9 months, and the analyst created a binary indicator (1 = YES, 0 = NO) of which customers were cross-sold as the target variable. The remaining independent attributes are based on the snapshot view of the customer’s data as of the time of service at the CoE. What modeling approaches would you use? What would your output to the business look like? How would you measure the effectiveness of your solutions? Based on your analysis, what recommendations and or insights do you have for the business?

data和data dictionary放在我的google drive上，link如下

https://drive.google.com/file/d/ ... F1/view?usp=sharing
https://drive.google.com/file/d/ ... ho/view?usp=sharing
https://drive.google.com/file/d/ ... up/view?usp=sharing

这是一个典型的imbalance data, target = 1的record只有 2%, 另外很多attributes都有不少missing data, 这些columns 既有categoricao的也有numerical的. 我是用random forecast 做的，但数据处理的思路应该是错了， 大家一般是怎么code missing data part的， 尤其是predictor比较多而且categoricao和numerical predictor都有的情况 （125 predictors in this case)

import pandas as pd
import numpy as np

data = pd.read_csv('Data.txt',sep="|")

def myfillna(series):
    if series.dtype is pd.np.dtype(int):
        return series.fillna(mode(series))
    elif series.dtype is pd.np.dtype(float):
        return series.fillna(series.median())
    elif series.dtype is pd.np.dtype(object):
        return series.fillna(0)
    else:
        return series

#text cateforical to dummy
data1 = pd.get_dummies(data, columns=['product'])
data2 = pd.get_dummies(data1, columns=['MAJOR_CREDIT_CARD_LIF'], dummy_na=True)

#drop columns with no data
data3 = data2.dropna(axis=1,how='all')

#impute missing data. If column type is integer, I will fill with mode. if column type is float, fill with median.
data4 = data3.apply(myfillna)

省略random forest部分, 这个比较standard.


大神们讨论讨论? 我当时答了很多种model的不同方案但 总觉得最开始的数据处理不好

imbalance data 一般先要over 或者 under sampling吧

licewolf 发表于 2019-6-29 11:26
imbalance data 一般先要over 或者 under sampling吧

很多oversampling算法不支持 missing data, 比如SMOTE, 要先把missing data的问题解决了才能接着用oversampling

#impute missing data. If column type is integer, I will fill with mode. if column type is float, fill with median.

====
为什么整数用众数而非中位数填充？另外lz是不是要考虑数据丢失的比例。
考虑一个极端的例子，一组数据30%是1，20%是2，10%是3，剩下的丢失，如果直接用众数/中位数填充会导致比例失调，应该从没有丢失的数据中random sampling从而填充

novavistamn 发表于 2019-6-30 10:14
#impute missing data. If column type is integer, I will fill with mode. if column type is float, fil ...

我是笼统地把整数attribute都当成categorical variable， 所以用mode. 带小数的都用median. 方法很粗糙，但是对于几百个columns， 不可能一个一个去看来决定用median还是Mode。
总觉得是有更好的办法，这是限时答题，assume我应该有个系统的方法而不是一个个去看