TY - JOUR
T1 - Context vectors are reflections of word vectors in half the dimensions
AU - Assylbekov, Zhenisbek
AU - Takhanov, Rustem
N1 - Funding Information:
The work of Zhenisbek Assylbekov has been funded by the Committee of Science of the Ministry of Education and Science of the Republic of Kazakhstan, contract # 346/018-
PY - 2019/9/1
Y1 - 2019/9/1
N2 - This paper takes a step towards the theoretical analysis of the relationship between word embeddings and context embeddings in models such as word2vec. We start from basic probabilistic assumptions on the nature of word vectors, context vectors, and text generation. These assumptions are supported either empirically or theoretically by the existing literature. Next, we show that under these assumptions the widely-used word-word PMI matrix is approximately a random symmetric Gaussian ensemble. This, in turn, implies that context vectors are reflections of word vectors in approximately half the dimensions. As a direct application of our result, we suggest a theoretically grounded way of tying weights in the SGNS model.
AB - This paper takes a step towards the theoretical analysis of the relationship between word embeddings and context embeddings in models such as word2vec. We start from basic probabilistic assumptions on the nature of word vectors, context vectors, and text generation. These assumptions are supported either empirically or theoretically by the existing literature. Next, we show that under these assumptions the widely-used word-word PMI matrix is approximately a random symmetric Gaussian ensemble. This, in turn, implies that context vectors are reflections of word vectors in approximately half the dimensions. As a direct application of our result, we suggest a theoretically grounded way of tying weights in the SGNS model.
UR - http://www.scopus.com/inward/record.url?scp=85075469760&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85075469760&partnerID=8YFLogxK
U2 - 10.1613/jair.1.11368
DO - 10.1613/jair.1.11368
M3 - Article
AN - SCOPUS:85075469760
VL - 66
SP - 225
EP - 242
JO - Journal of Artificial Intelligence Research
JF - Journal of Artificial Intelligence Research
SN - 1076-9757
ER -