CONVERGENCE OF THE PARTITION FUNCTION IN THE STATIC WORD EMBEDDING MODEL

K. Mynbaev; Zh Assylbekov

doi:10.32523/2077-9879-2022-13-4-70-81

CONVERGENCE OF THE PARTITION FUNCTION IN THE STATIC WORD EMBEDDING MODEL

K. Mynbaev, Zh Assylbekov

Research output: Contribution to journal › Article › peer-review

Abstract

We develop an asymptotic theory for the partition function of the word embeddings model word2vec. The proof involves a study of properties of matrices, their determinants and distributions of random normal vectors when their dimension tends to infinity. The conditions imposed are mild enough to cover practically important situations. The implication is that for any word i from a vocabulary W, the context vector c_i is a reflection of the word vector w_i in approximately half of the dimensions. This allows us to halve the number of trainable parameters in static word embedding models.

Original language	English
Pages (from-to)	70-81
Number of pages	12
Journal	Eurasian Mathematical Journal
Volume	13
Issue number	4
DOIs	https://doi.org/10.32523/2077-9879-2022-13-4-70-81
Publication status	Published - 2022

Keywords

asymptotic distribution
neural networks
partition function
word embeddings
word2vec

ASJC Scopus subject areas

General Mathematics

Access to Document

10.32523/2077-9879-2022-13-4-70-81

Cite this

@article{e00c69273e434141a85ef6a82f413a09,

title = "CONVERGENCE OF THE PARTITION FUNCTION IN THE STATIC WORD EMBEDDING MODEL",

abstract = "We develop an asymptotic theory for the partition function of the word embeddings model word2vec. The proof involves a study of properties of matrices, their determinants and distributions of random normal vectors when their dimension tends to infinity. The conditions imposed are mild enough to cover practically important situations. The implication is that for any word i from a vocabulary W, the context vector ci is a reflection of the word vector wi in approximately half of the dimensions. This allows us to halve the number of trainable parameters in static word embedding models.",

keywords = "asymptotic distribution, neural networks, partition function, word embeddings, word2vec",

author = "K. Mynbaev and Zh Assylbekov",

note = "Funding Information: Supported by the ISAAC (International Society for Analysis, its Applications and Computation) and by the Kazakhstan Mathematical Society Funding Information: The work is supported by the Nazarbayev University 091019CRP2109, PI Zhenisbek Assylbekov. Publisher Copyright: {\textcopyright} The L.N. Gumilyov Eurasian National University",

year = "2022",

doi = "10.32523/2077-9879-2022-13-4-70-81",

language = "English",

volume = "13",

pages = "70--81",

journal = "Eurasian Mathematical Journal",

issn = "2077-9879",

publisher = "L. N. Gumilyov Eurasian National University",

number = "4",

}

TY - JOUR

T1 - CONVERGENCE OF THE PARTITION FUNCTION IN THE STATIC WORD EMBEDDING MODEL

AU - Mynbaev, K.

AU - Assylbekov, Zh

N1 - Funding Information: Supported by the ISAAC (International Society for Analysis, its Applications and Computation) and by the Kazakhstan Mathematical Society Funding Information: The work is supported by the Nazarbayev University 091019CRP2109, PI Zhenisbek Assylbekov. Publisher Copyright: © The L.N. Gumilyov Eurasian National University

PY - 2022

Y1 - 2022

N2 - We develop an asymptotic theory for the partition function of the word embeddings model word2vec. The proof involves a study of properties of matrices, their determinants and distributions of random normal vectors when their dimension tends to infinity. The conditions imposed are mild enough to cover practically important situations. The implication is that for any word i from a vocabulary W, the context vector ci is a reflection of the word vector wi in approximately half of the dimensions. This allows us to halve the number of trainable parameters in static word embedding models.

AB - We develop an asymptotic theory for the partition function of the word embeddings model word2vec. The proof involves a study of properties of matrices, their determinants and distributions of random normal vectors when their dimension tends to infinity. The conditions imposed are mild enough to cover practically important situations. The implication is that for any word i from a vocabulary W, the context vector ci is a reflection of the word vector wi in approximately half of the dimensions. This allows us to halve the number of trainable parameters in static word embedding models.

KW - asymptotic distribution

KW - neural networks

KW - partition function

KW - word embeddings

KW - word2vec

UR - http://www.scopus.com/inward/record.url?scp=85150924400&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85150924400&partnerID=8YFLogxK

U2 - 10.32523/2077-9879-2022-13-4-70-81

DO - 10.32523/2077-9879-2022-13-4-70-81

M3 - Article

AN - SCOPUS:85150924400

SN - 2077-9879

VL - 13

SP - 70

EP - 81

JO - Eurasian Mathematical Journal

JF - Eurasian Mathematical Journal

IS - 4

ER -

CONVERGENCE OF THE PARTITION FUNCTION IN THE STATIC WORD EMBEDDING MODEL

Abstract

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this