有没有小伙伴有兴趣一起学习 machine learning

目前计划是每周列计划学习相关内容，做笔记打卡。有没有兴趣一起的？

补充内容 (2021-1-25 22:28):
并不是入门 ML，每个人学的也可以不一样，只是起到互相监督互相分享笔记的作用，有兴趣可以私信我

补充内容 (2021-1-25 23:47):
大致的计划是这样：每周大家列出自己的计划，然后【写学习笔记】互相分享。重点在于要做学习笔记（可以是文本笔记或者 代码，jupyter notebook 等等）

补充内容 (2021-1-26 10:13):
有兴趣的可以直接私信我，我不知道为什么不能私信

上过课，目前继续自学，感觉还是要找一些project上手才能理解得细致扎实。有推荐的github repo吗？

### learning important features: DeepLIFT
. 1point3acres- some approaches for identifying input features
    - perturbation-based approaches
        - perturb one feature, see change of output
        - drawback
            - computationally expensive (need to try all features)
            - susceptible to **saturation problem** in DL: for some functions, change in some input has little effect on output, and may mistakenly assume that this feature is not important
    - backpropagation-based approaches
        - idea: start with output, assign importance score to layer connected to the output layer, then propagate the importance score all the way back to inputs
        - drawback: depends on gradient and may have **vanishing gradient** issue
- DeepLIFT
    - high-level idea: compute difference of output from **reference point** (basically doing finite difference, not infinitesimal difference) so that importance score can be propagated back
        - reference point: depends on domain knowledge (e.g. for MNIST images, define all zeros as reference in input space, then propagate forward to get references for all later layers)
    - basics: let $\Delta t$ be diff-from-ref for target neuron $t$, and assign **contribution score** (how? see different rules) of input neuron $x_i$ to be $C_{\Delta x_i\Delta t}$ such that $\sum_i C_{\Delta x_i\Delta t}=\Delta t$.  We also define multiplier $$m_{\Delta x_i\Delta t}=\frac{C_{\Delta x_i\Delta t}}{\Delta x_i}$$.  With these two definitions, we see that it enables chain rule and backpropagation
    $$m_{\Delta x_i\Delta z}=\sum_j m_{\Delta x_i\Delta y_j}m_{\Delta y_j\Delta z}$$
    - separation of positive and negative contributions: $\Delta x_i=\Delta x_i^+ + \Delta x_i^-$, which is useful in some cases
    - linear rules for assigning contributions scores: for $y=b+\sum_i w_i x_i, \Delta y=\sum_i w_i\Delta x_i$, define $$C_{\Delta x_i^{\pm}\Delta y_i^{+}}=1\{w_i\Delta x_i>0\}w_i\Delta x_i^{\pm}\\C_{\Delta x_i^{\pm}\Delta y_i^{-}}=1\{w_i\Delta x_i<0\}w_i\Delta x_i^{\pm}$$, multiplier can be defined similarly
    - nonlinear rules
        - rescale rule: assume contribution of $\Delta y^+$ **only comes from $\Delta x^+$**.  $$C_{\Delta x^+\Delta y^+}=\frac{\Delta y}{\Delta x}\Delta x^+=\Delta y^+$$, therefore $$m_{\Delta x^{\pm}\Delta y^{\pm}}=\frac{\Delta y}{\Delta x}=m_{\Delta x\Delta y}$$, note that rescale rule fails for some problems
        - RevealCancel rule: $$\Delta y^+=\frac{1}{2}\left(f(x^0+\Delta x^+)-f(x^0)\right)\\+\frac{1}{2}\left(f(x^0+\Delta x^-+\Delta x^+)-f(x^0+\Delta x^-)\right)$$, and we have $$\Delta y=\Delta y^++\Delta y^-=f(x^0+\Delta x^-+\Delta x^+)-f(x^0)$$
. 1point 3acres     - possible implementation in PyTorch
        - use `hooks` (e.g. [here](https://github.com/pytorch/captu ... e/deep_lift.py#L546)) to override gradient operators, thus support a wide variety of architectures (as is explained in README [here](https://github.com/kundajelab/deeplift))
- ref
    - https://www.youtube.com/watch?v= ... 6KPGKML&index=6
    - https://arxiv.org/pdf/1704.02685.pdf
### model complexity and bias-variance tradeoff for neural networks
- bias-variance tradeoff for neural networks
    - for classical ML models, there is bias-variance tradeoff: as model complexity increases, bias goes down while variance goes up.  But for neural networks, **variance does not go up with network width**.  This behavior can be explained by its two components: **variance due to optimization** increases in under-parameterized regime and decreases in **over-parameterized regime**, while **variance due to sampling** plateaus and remains constant
    - ref
        - https://arxiv.org/abs/1810.08591
        - https://arxiv.org/abs/2002.11328
- double-descent curve
    - classical ML theory fails to explain the generalization properties of neural network models containing millions of parameters (in over-parameterized regime).  With these networks with super high complexity, one can often reach a lower error than the minimum of the networks in the **under-parametrized regime**, despite the perfect training accuracy.
    - ref
        - https://medium.com/@LightOnIO/be ... -curve-18b6d9810e1b

### normalizing flows

- idea
    - for variable $z$ satisfying following distribution $z\sim p_\theta(z)$, let $x$ be a bijective transformation of $z$: $$x=T(z)=f_k\circ...\circ f_2\circ f_1(z)$$, this normalizing flow is useful for constructing **generative models**, which generates $x$ that satisfies a **specific distribution** from latent variable $z$ (which usually satisfies a simpler distribution such as Gaussian), estimation of log likelihood of $x$ is given by $$\log p(x)=\log p_\theta(z)+\log |\text{det}(\nabla_zz)|^{-1}$$, which is an estimation of **exact log-likelihood** (unlike VAE which is just a lower bound).  This log likelihood can be trained with gradient descent with a series of invertible layers.  To make training efficient, we want to make both **inverse and Jacobian** computationally cheap
    - masked autoregressive flows
        - triangular maps: let $T_i(=x_i)$ be i-th component (not i-th transform function), and $T_i=T_i(z_1, ...,z_i)$ **only depends on first $i$ components of $z$**, now gradient is a triangular matrix and is easy to compute
        - theorem: there always exists a unique increasing triangular map that transforms a source density to a target density
        - masked autoregressive flows: stack multiple affine-transformation triangular maps with **different ordering** (so that if a specific dependency ordering cannot model target distribution well, other ordering may be able to do so)
    - other invertible layers
        - real-NVP and NICE
        - neural autoregressive flows: similar to masked autoregressive flows, but use general invertible neural networks, not just affine transformations, to be more expressive
        - sum-of-squares polynomial flows
        - Glow

- ref
    - https://www.youtube.com/watch?v=3KUvxIOJD0k
    - https://blog.evjang.com/2018/01/nf1.html (includes implementation in tensorflow distributions)
    - https://lilianweng.github.io/lil ... erative-models.html

不是入门ML了，不过我在学statistics，并且打算读那本经典的ML书，the elements of statistical learning。有兴趣可以私信我或者回复我让我私信你。

betameow96 发表于 2021-1-25 13:12
不是入门ML了，不过我在学statistics，并且打算读那本经典的ML书，the elements of statistical learning。 ...

并不是入门 ML，每个人学的也可以不一样，只是起到互相监督互相分享笔记的作用，有兴趣可以私信我（我好像不能私信。。。

一样想学，上过本科level的ML课但是感觉很多东西没有理解，求推荐project based的课程

dcyjff21 发表于 2021-1-25 22:40
一样想学，上过本科level的ML课但是感觉很多东西没有理解，求推荐project based的课程

kaggle 上有一些项目，github 上也有不少，有兴趣可以看看

feedback transformer
    - motivation
        - RNN can look at inputs in sequence, but cannot be trained in parallel, while transformer looks at all inputs in a sequence at once, and doing this may **lose some sequential information**, e.g. if we use transformer to process a program source code, it may be difficult (requires many layers to do multi-step reasoning) to keep track of order in which the value of a variable is updated
        - technically, the information for a transformer can flow towards up right (lowest is input layer, "towards up right" due to causal masking), there is no connection among nodes at the same level (otherwise cannot train in parallel)
    - architecture
        - each node at timestep t can access information at (t-1) for nodes **at all levels** (including nodes at levels higher than this node), we no longer have parallel training, but
            - we have **better sequential information encoded in the model**, which allows more complicated multi-step reasoning with relatively shallow models
            - we can use **all-level representation** at all previous steps together with representations at current step
        - to reduce model complexity, they use a **memory vector** that summarizes representations of all layers at step t, then any node at step t can look at memory vectors at **all previous steps** (so still have **long-term information access**, unlike RNN where information can only propagate to next step)
    - training
        - although not fully parallelizable, sharing of key-value pairs improves training speed
    - ref
        - https://arxiv.org/abs/2002.09402
        - https://www.youtube.com/watch?v=zdb8MM94A5c

- nonlinear shrinkage with cross-validation
    - method
        - they use **spectrum correction** method to estimate cov matrix given samples, meaning that first compute sample cov from data, then **keep its eigenvectors unchanged**, and **modify only eigenvalues**
        - the state-of-the-art is Ledoit and Wolf's **nonlinear shrinkage** method, but it is complicated and computationally expensive.  This paper uses CV to achieve similar performance with much lower cost


        - given population (theoretical) cov $C$, and sample eigenvectors $\{\hat{v}_i\}$.  The optimal corrected eigenvalues are given by minimizing $$\gamma^*=\argmin_\gamma ||C-\sum_i \gamma_i\hat{v}_i\hat{v}_i^T||_F^2$$, we have $\gamma_i^*=\hat{v}_i^T C \hat{v}_i$, since $C$ is unknown, we cannot compute it this way
        - with LOO CV, we compute $\hat{C}^{(t)}$ to be sample cov using data excluding $x_t$, corresponding eigenvectors are $\{\hat{v}_i^{loo(t)}\}$, the LOO-CV estimator is
        $$\hat{\gamma}_i^{loo-cv} = \frac{1}{n}\sum_{t=1}^n (\hat{v}_i^{loo(t)T}x_t)^2$$, since $\hat{v}_i^{loo(t)}$ and $x_t$ are independent, it can be proved that it is an unbiased estimator of $$\left(\hat{v}_i^{loo(t)T}C\hat{v}_i^{loo(t)}\right)$$
        - two issues with LOOCV above
            - could be noisy in some cases (Fig. 1), Fix: use K-fold CV
            - resulting eigenvalues are not ordered, Fix: apply **isotonic regression** on top of unordered estimates
    - ref: 1611.00798