Incremental Sense Weight Training for In-Depth Interpretation of Contextualized Word Embeddings

Xinyi Jiang, Zhengzhe Yang, Jinho D. Choi


Abstract

We present a novel online algorithm that learns the essence of each dimension in word embeddings. We first mask dimensions determined unessential by our algorithm, apply the masked word embeddings to a word sense disambiguation task (WSD), and compare its performance against the one achieved by the original embeddings. Our results show that the masked word embeddings do not hurt the performance and can improve it by 3%.

Venue / Year

Proceedings of the AAAI Conference on Artificial Intelligence: Student Abstract and Poster Program (AAAI:SAP) / 2020

Links

Anthology | Paper | Presentation | BibTeX