Paper 2511.15210
Unveiling Intrinsic Dimension of Texts: from Academic Abstract to Creative Story
- Published
- Nov 2025
- Research lab
- Independent
- Citations
- 1
- GitHub
- Not linked
01 In brief
Summary
This paper provides the first comprehensive study of intrinsic dimension (ID) in text representations, grounding it in interpretable properties through cross-encoder analysis, linguistic features, and sparse autoencoders (SAEs).
The authors establish three key findings: (1) ID is complementary to entropy-based metrics, as after controlling for length, the two are uncorrelated, with ID capturing geometric complexity orthogonal to prediction quality; (2) ID exhibits robust genre stratification, with scientific prose showing low ID (~8), encyclopedic content medium ID (~9), and creative/opinion writing high ID (~10.5) across all tested models; (3) using SAEs, they identify causal features where scientific signals (formal tone, report templates, statistics) reduce ID, while humanized signals (personalization, emotion, narrative) increase it, confirmed by steering experiments.
The study uses the COLING dataset (172,750 texts after filtering) and PHDIM as the primary ID estimator, validated against MLE, TLE, and TwoNN.
Linguistic analysis shows ID increases with lexical diversity and decreases with sentence-level repetition, while syntactic metrics are uninformative for well-formed texts.
The findings suggest that for contemporary LLMs, scientific writing is representationally simple, while fiction and opinionated writing require additional degrees of freedom, with practical implications for evaluation and training data selection.
02 From the paper
Abstract
Intrinsic dimension (ID) is an important tool in modern LLM analysis, informing studies of training dynamics, scaling behavior, and dataset structure, yet its textual determinants remain underexplored. We provide the first comprehensive study grounding ID in interpretable text properties through cross-encoder analysis, linguistic features, and sparse autoencoders (SAEs). In this work, we establish three key findings. First, ID is complementary to entropy-based metrics: after controlling for length, the two are uncorrelated, with ID capturing geometric complexity orthogonal to prediction quality. Second, ID exhibits robust genre stratification: scientific prose shows low ID (~8), encyclopedic content medium ID (~9), and creative/opinion writing high ID (~10.5) across all models tested. This reveals that contemporary LLMs find scientific text "representationally simple" while fiction requires additional degrees of freedom. Third, using SAEs, we identify causal features: scientific signals (formal tone, report templates, statistics) reduce ID; humanized signals (personalization, emotion, narrative) increase it. Steering experiments confirm these effects are causal. Thus, for contemporary models, scientific writing appears comparatively "easy", whereas fiction, opinion, and affect add representational degrees of freedom. Our multi-faceted analysis provides practical guidance for the proper use of ID and the sound interpretation of ID-based results.