- 来源：2104.01767v1
- 任务：从预训练模型得到最好的句子表示
- 结论：
- averaging all token representations con- sistently induces better sentence representations than using the [CLS] token embedding.
- combining the embeddings of the bottom layer and the top layer outperforms that using top two layers.
- normalizing sentence embeddings with whitening algorithm consistently boosts the performance. 这里 whitening 是指将 embedding 矩阵搞成协方差矩阵为对称矩阵的形式，即 $\hat{E} = (E-m)UD^{-\frac{1}{2}}$