TY - GEN
T1 - Squashed Shifted PMI Matrix
T2 - 33rd Australasian Joint Conference on Artificial Intelligence, AI 2020
AU - Assylbekov, Zhenisbek
AU - Jangeldin, Alibi
N1 - Funding Information:
Acknowledgements. This work is supported by the Nazarbayev University faculty-development competitive research grants program, grant number 240919FD3921. The authors would like to thank Zhuldyzzhan Sagimbayev for conducting preliminary experiments for this work, and anonymous reviewers for their feedback.
Publisher Copyright:
© 2020, Springer Nature Switzerland AG.
PY - 2020
Y1 - 2020
N2 - We show that removing sigmoid transformation in the skip-gram with negative sampling (SGNS) objective does not harm the quality of word vectors significantly and at the same time is related to factorizing a squashed shifted PMI matrix which, in turn, can be treated as a connection probabilities matrix of a random graph. Empirically, such graph is a complex network, i.e. it has strong clustering and scale-free degree distribution, and is tightly connected with hyperbolic spaces. In short, we show the connection between static word embeddings and hyperbolic spaces through the squashed shifted PMI matrix using analytical and empirical methods.
AB - We show that removing sigmoid transformation in the skip-gram with negative sampling (SGNS) objective does not harm the quality of word vectors significantly and at the same time is related to factorizing a squashed shifted PMI matrix which, in turn, can be treated as a connection probabilities matrix of a random graph. Empirically, such graph is a complex network, i.e. it has strong clustering and scale-free degree distribution, and is tightly connected with hyperbolic spaces. In short, we show the connection between static word embeddings and hyperbolic spaces through the squashed shifted PMI matrix using analytical and empirical methods.
KW - Complex networks
KW - Hyperbolic geometry
KW - PMI
KW - Word vectors
UR - http://www.scopus.com/inward/record.url?scp=85097645140&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85097645140&partnerID=8YFLogxK
U2 - 10.1007/978-3-030-64984-5_26
DO - 10.1007/978-3-030-64984-5_26
M3 - Conference contribution
AN - SCOPUS:85097645140
SN - 9783030649838
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 336
EP - 346
BT - AI 2020
A2 - Gallagher, Marcus
A2 - Moustafa, Nour
A2 - Lakshika, Erandi
PB - Springer Science and Business Media Deutschland GmbH
Y2 - 29 November 2020 through 30 November 2020
ER -