WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach

  • 来源:2104.01767v1
  • 任务:从预训练模型得到最好的句子表示
  • 结论:
    • averaging all token representations con- sistently induces better sentence representations than using the [CLS] token embedding.
    • combining the embeddings of the bottom layer and the top layer outperforms that using top two layers.
    • normalizing sentence embeddings with whitening algorithm consistently boosts the performance. 这里 whitening 是指将 embedding 矩阵搞成协方差矩阵为对称矩阵的形式,即 $\hat{E} = (E-m)UD^{-\frac{1}{2}}$