Introduction

Word embeddings represent word meanings as points in a high-dimensional continuous space. An intriguing finding is that certain principal components or directions in these spaces correspond to broad, human-interpretable semantic dimensions. Classic psycholinguistic research by Osgood et al. (1957) identified valence (good–bad), arousal (excited–calm), and dominance (powerful–weak) as the primary axes of connotative meaning. Recent studies suggest that modern data-driven embeddings independently rediscover similar dimensions. In this report, we examine how robustly these semantic axes emerge across different embedding methods – from early thesaurus-based graphs to neural embeddings like Word2Vec, GloVe, and transformer-based models. We also discuss analytical techniques for uncovering these dimensions, and enumerate a set of prominent semantic components (beyond valence and arousal) that have been observed in English word embedding spaces.

Early Semantic Maps and Principal Axes

Before the era of neural embeddings, researchers constructed semantic maps from lexical graphs (thesauri and dictionaries). These maps often revealed that the top dimensions of variation aligned with intuitive meaning contrasts. For example, Samsonovich & Ascoli (2010) built a low-dimensional semantic space by integrating synonym and antonym relations from dictionaries. Strikingly, the first three principal components of their semantic map had clear meanings: valence (good vs. bad), arousal (calm vs. excited), and a third dimension they labeled “freedom” (open vs. closed) (
Principal Semantic Components of Language and the Measurement of Meaning - PMC). In other words, words were arranged such that the primary axis separated positive concepts from negative ones, the second separated exciting or energetic concepts from dull or calm ones, and the third separated notions of openness/liberty from closedness or constraint (which the authors note corresponds to a dominance or potency dimension) (
Principal Semantic Components of Language and the Measurement of Meaning - PMC). These dimensions emerged robustly: the same three were found using two different lexicons (WordNet and the Microsoft Word thesaurus) and even across multiple languages (English, French, German, Spanish) (
Principal Semantic Components of Language and the Measurement of Meaning - PMC). This early evidence suggested that affective and experiential qualities form the fundamental coordinates of semantic space, even when derived from human-curated lexical resources.

Not only do thesaurus-based representations exhibit these axes, but simple spatial layouts of word graphs show similar patterns. Force-directed graphs of synonym networks, for instance, tend to cluster positive words separately from negative ones, indicating an underlying valence dimension. Overall, early graph-based semantic representations provided a proof of concept that principal components of meaning correspond to recognizable scales like evaluative sentiment and intensity of experience.

Modern Neural Embeddings (Word2Vec, GloVe, BERT)

Contemporary embeddings learned from raw text (e.g. Word2Vec skip-gram, GloVe, or transformer models) encode semantic relationships in dense vectors. Researchers have applied dimensionality reduction and other analyses to these embeddings and found that human-relevant semantic dimensions likewise surface in these learned spaces. However, the ordering of these dimensions by explained variance can differ from the thesaurus case. In Word2Vec embeddings trained on large corpora, affective dimensions appear within the first dozen principal components rather than as the very first two (The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics | Psychonomic Bulletin & Review) (The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics | Psychonomic Bulletin & Review). For example, Hollis and Westbury (2017) analyzed the first eight principal components of a skip-gram (Word2Vec) model. They observed that one principal component strongly corresponded to valence (separating positive words like "splendid" and "vibrant" from negative or unpleasant words like "harmful" and "anus") and another to dominance (contrasting words implying lack of control like "desolate" or "hellish" with those suggesting being in control, e.g. "articulate", "compliment") (The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics | Psychonomic Bulletin & Review) (The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics | Psychonomic Bulletin & Review). In their analysis, valence was the 5th principal component and dominance was the 7th, showing that while these dimensions are clearly present, other variance factors come earlier (The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics | Psychonomic Bulletin & Review). Notably, they found concreteness (abstract vs. concrete) to be a very strong 2nd component, and word frequency to align with the 1st principal component (The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics | Psychonomic Bulletin & Review). The dominance of frequency in PC1 is likely because very frequent words (e.g. function words or generic terms) occupy a distinct region of embedding space. After controlling for such factors, the affective axes emerge prominently. In short, Word2Vec’s geometry encodes sentiment and experiential qualities similarly to the human semantic differential, even if these are not always the top-ranked PCs by variance (The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics | Psychonomic Bulletin & Review) (The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics | Psychonomic Bulletin & Review).

Other embedding algorithms show comparable patterns. Analyses of GloVe embeddings (which are also based on word co-occurrence statistics) report that a valence dimension can be identified that correlates with human sentiment ratings (2-components PCA representations of GloVe, base BERT, GoEmotion BERT,... | Download Scientific Diagram). Recent work by Zhang et al. (2023) compared GloVe with contextualized transformer embeddings, finding that all showed significant correlations with human valence, arousal, and dominance ratings (2-components PCA representations of GloVe, base BERT, GoEmotion BERT,... | Download Scientific Diagram). In their study, the authors took a large set of English words with known emotion norms and performed PCA on different embedding spaces. The first one or two principal components of each embedding type were meaningfully aligned with the VAD (Valence-Arousal-Dominance) axes, although the strength varied (with static embeddings like GloVe encoding valence more clearly than the base BERT model) (2-components PCA representations of GloVe, base BERT, GoEmotion BERT,... | Download Scientific Diagram). This suggests that even in the high-dimensional contextual embeddings of a transformer, there exist principal directions corresponding to these core semantic attributes, though they may be somewhat diluted across dimensions due to the model’s context-dependent nature.

Transformer-based models (like BERT) learn token embeddings that are context-sensitive, making direct analysis trickier. But one can examine the static embedding of a word (e.g. the first principal component of all its contextual instances, as done by Zhang et al. 2023 ()). Such studies have confirmed that BERT’s embedding space contains affective dimensions, but less starkly than static word2vec/GloVe embeddings (2-components PCA representations of GloVe, base BERT, GoEmotion BERT,... | Download Scientific Diagram). This is plausibly because transformers distribute semantic features across many dimensions and rely on context for disambiguation. Nonetheless, when averaging over many contexts, words with high valence (like “joyful” or “success”) cluster apart from those with low valence (“terrible”, “death”), indicating a latent valence axis. In sum, modern neural embeddings do encode human-interpretable dimensions: not only valence and arousal, but also others like concreteness, as evidenced by consistent findings across different models and corpora (The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics | Psychonomic Bulletin & Review).

One particularly comprehensive study (Westbury et al. 2025) extended the analysis to a larger word set and found that certain semantic categories reliably predict the values of the early principal components across models (The principal components of meaning, revisited - PubMed). Among the most predictive were affect-laden words (supporting the valence/arousal dimensions), personal pronouns (distinguishing 1st vs 2nd person contexts), concreteness (abstract vs concrete terms), and proper nouns like people and place names (The principal components of meaning, revisited - PubMed). The fact that these predictors generalized well even to embeddings trained on different corpora and with different algorithms (The principal components of meaning, revisited - PubMed) underscores the robustness of these semantic axes as fundamental organizing principles in English lexical semantics.

Methods for Uncovering Semantic Axes

Several analytical approaches shed light on these principal semantic dimensions in embeddings:

• Principal Component Analysis (PCA): This is the most direct method, used in many of the studies above. By reducing the embedding matrix and examining the top components, researchers can see which linguistic variables correlate with each component (The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics | Psychonomic Bulletin & Review). PCA is unsupervised and finds orthogonal directions of maximum variance, which often correspond to major semantic contrasts (though not always purely one semantic factor). For example, PCA on word2vec revealed distinct PCs for concreteness, valence, dominance, etc. (The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics | Psychonomic Bulletin & Review). One drawback is that some interpretable dimensions might not be among the very first components if they do not explain the largest variance; for instance, sentiment wasn’t PC1 in Word2Vec, as noted.

• Factor Analysis and Semantic Differential: This approach has a long history in psychology (e.g. Osgood’s factor analysis of word ratings). It treats the data as reflecting latent factors and can sometimes yield more interpretable (rotated) factors. In practice, applying factor analysis to embedding distances or similarity judgments can recover valence, arousal, dominance as factors, as was done with early semantic differential experiments. Modern implementations are similar to PCA in effect and often identify the same axes (with perhaps some rotation).

• Independent Component Analysis (ICA): Unlike PCA, ICA does not prioritize variance and seeks statistically independent directions. Recent research finds that ICA can produce even more interpretable axes from embeddings, treating each dimension as a latent semantic feature (Exploring Interpretability of Independent Components of Word Embeddings with Automated Word Intruder Test) (Exploring Interpretability of Independent Components of Word Embeddings with Automated Word Intruder Test). For example, Musil and Mareček (2022) showed that most independent components correspond to clear semantic groupings (and performed automated “word intrusion” tests to verify this) (Exploring Interpretability of Independent Components of Word Embeddings with Automated Word Intruder Test) (Exploring Interpretability of Independent Components of Word Embeddings with Automated Word Intruder Test). ICA has the advantage that no single component dominates; thus, even a nuance like “edibility” or “is an animal” can emerge as an independent feature rather than being suppressed by larger variance trends.

• Supervised Probing (Regression/Classification): Another method is to explicitly train a linear model to predict a known semantic property from the embeddings. For instance, one can learn a direction that best predicts human valence ratings (essentially a supervised probe). If the model succeeds with high accuracy, it confirms the embedding encodes that information in a linear subspace. Zhang et al. (2023) did this with a simple classifier for valence/arousal/dominance and found the embeddings are linearly separable on those dimensions to a notable degree (2-components PCA representations of GloVe, base BERT, GoEmotion BERT,... | Download Scientific Diagram) (2-components PCA representations of GloVe, base BERT, GoEmotion BERT,... | Download Scientific Diagram). Supervised probing has also been used for traits like formality or complexity of words, confirming that those latent traits can be extracted from embeddings () ().

• Semantic Axis by Seed Words: A more targeted approach identifies a dimension by using pairs of antonyms or seed words. For example, to find a gender axis, one might average the vector difference of (“she” – “he”), (“woman” – “man”), etc. (as in Bolukbasi et al. 2016). This method was famously used to debias embeddings, but it also demonstrated that a single direction can represent gender associations. Similar seed-based vectors have been used for formality (e.g. “casual” – “formal” seeds) and even physical properties like size or danger () (). The downside is that choosing seed words can be subjective, but when done carefully (or augmented with a few human ratings ()) it can yield clear interpretable axes.

In practice, these methods are complementary. PCA gives a broad picture of the dominant variations in meaning, while targeted probing or seed vectors can home in on specific known dimensions (even if they are not top variance components). The consistency of results across methods – for example, both PCA and supervised approaches identify valence and concreteness dimensions in word2vec (The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics | Psychonomic Bulletin & Review) – increases confidence that these are real, meaningful dimensions in the embedding space, not artifacts of any one technique. The use of multiple corpora and models (as in Westbury et al. 2025) further confirms the robustness of these semantic dimensions: regardless of how the word vectors are learned, words that differ in evaluative meaning or in concreteness tend to separate along particular directions in the space (The principal components of meaning, revisited - PubMed).

Major Interpretable Dimensions of Meaning in Embeddings

Beyond the basic valence (positivity) and arousal (intensity) axes, research has identified numerous other dimensions of meaning that emerge in English word embeddings. Table 1 below summarizes a set of prominent semantic dimensions found to be encoded in embedding spaces, along with a brief description of each. These dimensions are all human-interpretable: that is, each corresponds to a coherent semantic property or conceptual contrast. Many were initially documented as principal components in unsupervised analyses, or as meaningful directions obtained via the methods above. Notably, these dimensions extend our description of semantic space to qualities like power, concreteness, formality, and more, offering a deeper “feature list” for word meanings as captured by distributional models.

Dimension	Interpretation (Semantic Contrast)
Valence (Evaluation)	Positive vs. negative sentiment or evaluative meaning (Principal Semantic Components of Language and the Measurement of Meaning - PMC). One end of this axis contains words denoting pleasant, favorable, or good things, while the opposite end contains unpleasant, unfavorable, or bad things. This is a primary affective dimension distinguishing “good” vs. “bad” concepts.
Arousal (Activity)	High-energy, exciting or intense vs. low-energy, calm or dull (Principal Semantic Components of Language and the Measurement of Meaning - PMC). Words associated with agitation, intensity or liveliness (e.g. thrilling, furious) lie at one end, whereas words for serenity, boredom or inactivity (e.g. peaceful, sedate) lie at the other.
Dominance (Potency)	Powerful, controlling vs. weak, submissive or uncontrolled ([The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics
Concreteness	Abstract vs. concrete meaning ([The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics
Agency/Animacy	Agentive animate beings vs. inanimate or passive entities ([The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics
Gender Association	Female-associated vs. male-associated contexts ([The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics
Formality/Register	Informal or colloquial vs. formal or technical usage ([The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics
Complexity (AoA)	Basic, early-acquired words vs. advanced, learned-late words ([The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics
Size/Magnitude	Small or diminutive vs. large or grand in size/extent. This is a concrete property dimension: adjectives like tiny, miniature, slight occupy one end, whereas huge, gigantic, massive occupy the other. Even nouns and verbs can carry size connotations (e.g. mouse vs. elephant in context). Embeddings have been shown to encode such physical scalar properties ().
Named Entity (Properness)	Common/general vs. Proper (name-specific). This dimension separates generic concepts from specific names of people or places. On one side are proper nouns (personal names, city names, organizations), and on the other are common nouns and other words. For example, John, London, Microsoft vs. man, city, company. This reflects whether a word represents a unique entity or a class of entities. Studies found that names of people and places load strongly on certain principal components (The principal components of meaning, revisited - PubMed), distinguishing them from general vocabulary.

Table 1: Prominent semantic dimensions found in English word embedding spaces, with a brief description of the contrast each dimension represents. Each dimension corresponds to a direction in the embedding space along which words are organized by a particular aspect of meaning.

Conclusion

The evidence from a variety of embedding paradigms and analysis techniques converges on a clear message: word embeddings encode the same fundamental semantic dimensions that humans intuitively perceive. Dimensions such as valence (good–bad) and arousal (active–passive) are not only theoretical constructs but actually manifest as principal axes in both graph-based maps (
Principal Semantic Components of Language and the Measurement of Meaning - PMC) and neural embeddings (The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics | Psychonomic Bulletin & Review). Moreover, a host of other interpretable dimensions – including dominance, concreteness, gender, formality, size, agency, and more – can be identified as salient directions in the embedding space. These axes are remarkably robust across models and methods: whether one uses PCA on a Word2Vec matrix or probes a transformer, one finds similar semantic contrasts underlying the vector geometry (The principal components of meaning, revisited - PubMed) (2-components PCA representations of GloVe, base BERT, GoEmotion BERT,... | Download Scientific Diagram). This suggests that distributional learning from language captures deep regularities of meaning that are grounded in human experience (emotion, perception, social context, etc.). In practical terms, knowing that these principal components correspond to real-world semantics allows us to interpret and manipulate embeddings more effectively – for example, to adjust a text’s formality by moving along that axis, or to detect bias by examining the gender dimension. Ongoing research continues to refine how we extract and utilize these meaningful components, bringing us closer to truly interpretable word embeddings that align with human semantic knowledge.

Sources: The findings and examples above are supported by peer-reviewed studies and analyses of English embeddings, including Samsonovich & Ascoli (2010) (
Principal Semantic Components of Language and the Measurement of Meaning - PMC) (
Principal Semantic Components of Language and the Measurement of Meaning - PMC), Hollis & Westbury (2017) (The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics | Psychonomic Bulletin & Review) (The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics | Psychonomic Bulletin & Review), Westbury et al. (2025) (The principal components of meaning, revisited - PubMed), and recent work by Zhang et al. (2023) (2-components PCA representations of GloVe, base BERT, GoEmotion BERT,... | Download Scientific Diagram), among others. These sources collectively illustrate the emergence of human-interpretable dimensions in word embedding spaces and document the specific axes summarized in Table 1.