Word Embeddings and A La Carte Embeddings

Word embeddings map words to vectors so that a model can work with language numerically. A useful embedding places words used in similar contexts near one another: vectors for cat and dog, for example, should usually be more similar than vectors for cat and database. Similarity is commonly measured with cosine similarity.

How Distributional Embeddings Work #

The distributional hypothesis is the practical idea behind many embeddings: words that occur in similar contexts tend to have related meanings. Algorithms such as Word2Vec, GloVe, and fastText learn vector representations from large text collections by predicting nearby words or by factoring word-context statistics.

An embedding is not a dictionary definition. Its geometry reflects patterns in the training data, including ambiguity and bias. The vector for bank may mix the meanings of a financial institution and a riverbank unless the model is contextual.

Static and Contextual Representations #

Static models assign one vector to each vocabulary item. They are compact and useful for simple tasks, but cannot distinguish a word’s sense from its surrounding sentence.

Contextual language models produce a representation for each token occurrence. In a Transformer, the representation of bank changes with the rest of the input. That makes contextual embeddings better suited to many modern language tasks, although they cost more to compute.

A La Carte Embeddings #

A La Carte Embeddings is a method for inducing a vector for a new word or phrase from the contexts in which it appears. It is especially useful when a pretrained static embedding vocabulary does not contain a domain-specific term, a named entity, or a newly coined expression.

The method starts with a pretrained embedding space and learns a linear transformation from context vectors to target-word vectors. To embed an unseen term, collect its surrounding words, average their known embeddings, and apply the learned transformation. This is usually more reliable than using the raw context average because the transformation corrects systematic differences between a context representation and a word representation.

Practical Considerations #

Use a corpus from the domain where the new term appears, exclude the term itself from its context when appropriate, and collect enough varied examples to reduce noise. A La Carte Embeddings can extend a static model economically, but it cannot invent evidence that is absent from the corpus or resolve every ambiguous usage.