TY - JOUR
T1 - Hyperbolic Embedding for Finding Syntax in BERT
AU - Auyespek, Temirlan
AU - Mach, Thomas
AU - Assylbekov, Zhenisbek
N1 - Publisher Copyright:
© 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org)
PY - 2022
Y1 - 2022
N2 - Recent advances in natural language processing have improved our understanding of what kind of linguistic knowledge is encoded in modern word representations. For example, methods for testing the ability to extract syntax trees from a language model architecture were developed by Hewitt and Manning (2019)-they project word vectors into Euclidean subspace in such a way that the corresponding squared Euclidean distance approximates the tree distance between words in the syntax tree. This work proposes a method for assessing whether embedding word representations in hyperbolic space can better reflect the graph structure of syntax trees. We show that the tree distance between words in a syntax tree can be approximated well by the hyperbolic distance between corresponding word vectors.
AB - Recent advances in natural language processing have improved our understanding of what kind of linguistic knowledge is encoded in modern word representations. For example, methods for testing the ability to extract syntax trees from a language model architecture were developed by Hewitt and Manning (2019)-they project word vectors into Euclidean subspace in such a way that the corresponding squared Euclidean distance approximates the tree distance between words in the syntax tree. This work proposes a method for assessing whether embedding word representations in hyperbolic space can better reflect the graph structure of syntax trees. We show that the tree distance between words in a syntax tree can be approximated well by the hyperbolic distance between corresponding word vectors.
KW - BERT
KW - Poincaré ball
KW - Structural probe
UR - http://www.scopus.com/inward/record.url?scp=85124380026&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85124380026&partnerID=8YFLogxK
M3 - Conference article
AN - SCOPUS:85124380026
SN - 1613-0073
VL - 3078
SP - 58
EP - 64
JO - CEUR Workshop Proceedings
JF - CEUR Workshop Proceedings
T2 - 2021 International Conference of the Italian Association for Artificial Intelligence, AIxIA 2021 DP
Y2 - 1 December 2021 through 3 December 2021
ER -