在职个人战拖打卡DataCamp

帮昨天打卡完成了intermediate importing data in python~万变不离其宗，复习了一下BeautifulSoup回想写coursera capstone的时候那个痛苦劲真是因为不懂html结构。。。

打卡data cleaning~ 原来小老板之前用的fuzzywuzzy在这里学到了

打卡完成了Exploratory Data Analysis in Python～放假的时候完成了Working with Dates and Times和Writing Functions. Decorator的嵌套之类的还是有一点搞...这个career track完成超过一半啦，感觉确实帮助查缺补漏了很多！任重道远～加油加油！

打卡昨天完成了Analyze police activities~

5.30 打卡完成了Statistical Thinking in Python I
上学的时候distribution没有很好地理解，没想到DataCamp给了一个系统性复习，重温了Poisson的定义，从visualization的角度了解了怎么看数据的分布是否符合理论上binomial或者poisson的分布～
用loop 来进行sampling真的好帮助理解啊，为什么上学的时候统计没有和编程一起教～
ecdf() function主要是从一个sample里得出分布曲线的x和y. x是sample里各个值排列过后的序列，y是1/n到1的等差序列，用来记录cumulative probability

5/31 打卡Statistical Thinking in Python (Part 2) 前三章

这周末的游行真的让我很难过。纵观美国对于非洲族裔的压迫，从slavery到segregation，到mass incarceration，可以说在美国的非洲族裔是全世界最受压迫的少数族裔了。我绝对不支持暴力和打砸抢，可是什么时候这个社会才可以给少数族裔更多话语权，让代表他们的人真正进入执法制法的权利中心，理智地按宪法讨论法章？

06/01 打卡完成Statistical Thinking in Python (Part 2)～总结一些概念如下：
1) ECDF 用来做EDA，有些差别通过这个图可以看出大致mean和variation的差异
2）bootstrap sampling 用来做parameter estimation和计算confidence interval，bootstrap replicates计算test statistic；pairs bootstrap对index进行resample，再用同一index resample datasets
3）shifted means用于hypothesis testing，测试两个datasets的mean是否不同，用bootstrap sampling -- 需concatenate两个datasets来算
4）permutation用于hypothesis testing，测试不同datasets的distribution是否相同
5) EDA无法做conclusion，it is impossible to say if this is a real effect or just due to noise without computing a p-value.

06/02 今天完成了Handwashing project
1) 同时plot两个dataset的简单方法：创建ax object
ax = df1.plot(x="col_a", y="col_b",
              label="df1")
df2.plot(x="col_a", y="col_b",
         label="df2", ax=ax)
This is how you would add a label to the y-axis:
ax.set_ylabel("A label")

2) 重温import data + parse dates:
df = pd.read_csv('test.csv', parse_dates=['datetime'])

3) bootstrap sampling:
size应是len(selected column)

06/03 打卡Supervised Learning with Scikit Learn前两章～
1）连环画图：Be sure to begin your plotting statements for each figure with plt.figure() so that a new figure will be set up. Otherwise, your plots will be overlayed onto the same figure.
2）extract predictor and feature columns: df[''].values; df.drop('', axis=1).values
3）building correlation heatmaps: sns.heatmap(df.corr(), square=True, cmap='RdYlGn')
4）%timeit: time the computation

06/04 打卡完成了Supervised Learning with Scikit Learn第三章～
1）复习confusion matrix：
accuracy = (TP+TN) / (TP+TN+FP+FN)
precision = (TP) / (TP+FP) --> predicted positives; high precision = the predicted positives are mostly correct
recall = (TP) / (TP+FN) --> actual positives; high recall = the actual positives are mostly captured
f1 score = harmonic mean of precision and recall

2）Evaluate models:
• classification report, confusion matrix
• ROC - visually evaluate models
• precision-recall curve
• ROC score, roc_auc_score
• cross_val_score
• GridSearchCV

3）hold-out set in practice
Methodology: Create the hyperparameter grid -->Instantiate a logistic regression classifier --> Create training and test sets --> Tune the hyperparameters on the training set using GridSearchCV with 5-folds --> Print the best parameter and best score obtained from GridSearchCV by accessing the best_params_ and best_score_ attributes of logreg_cv

4）hyperparameter tuning
parameters in linear regressions
C, penalty in Logistic Regression
max_features, max_depth, and min_samples_leaf in Decision Tree
l1_ratio in elastic net models