Why do distances between random points become almost equal in high dimensions?

Squared distance $D^2$ is the sum of $d$ of these, one per coordinate, and the terms are independent, so the means add and the variances add. The variance of a single term needs $E[(x-y)^4]$, which for the difference of two uniform coordinates is $16/15$; subtracting the square of the mean, $4/9$, leaves $28/45$, a little over $0.6$.

Adding coordinates does push points apart, so the intuition is right about the average and wrong about what a search uses. Every pair gets pushed by roughly the same amount, and a pair only stands out through the imbalance in its contributions. That imbalance is itself a sum of $d$ independent terms, so it grows like $\sqrt{d}$, while the distance it perturbs grows like $d$; the ratio falls away. More coordinates therefore make points farther apart and less distinguishable at the same time, and it is the second that bites. The nearest neighbour of a query in three hundred coordinates sits within a few per cent of a randomly chosen point, so a ranking built on distance is reading a very thin signal.

The same arithmetic applies to angles. The cosine of two random directions is a normalised sum of products, and each product has mean zero, so the sum sits near zero. With coordinates uniform on $[-1,1]$ each product $a_ib_i$ has variance $1/9$, so the sum of $d$ of them has standard deviation $\sqrt{d}/3$, while the normalising lengths multiply to about $d/3$.

At three hundred coordinates that is $0.058$, an angle of $86.7$ degrees, or a little over three degrees off perpendicular. Two random directions in such a space are very nearly at right angles, which is the geometric restatement of the distance fact: the squared distance between unit vectors is $2 - 2\cos$, and a cosine pinned near zero pins the distance near $\sqrt{2}$. The good news in the same statement is capacity. Because near-perpendicular directions are cheap, a few hundred coordinates hold far more than a few hundred nearly independent directions, which is what lets a compact vector stand for a distinct notion.

All of this is stated in the number of independent contributions, not the number of stored coordinates. Coordinates that are strongly correlated, or heavy tailed, or that share a large common component, reduce the effective count and the concentration weakens. Real learned vectors are the standard example: they occupy a narrow cone rather than spreading over the sphere, so two unrelated items already score around $0.4$ rather than near zero, and the gap between related and unrelated narrows to about $0.45$. The ordering still holds and retrieval still works, but a threshold of $0.7$ means nothing outside the collection it was calibrated on, and a method tuned on low-dimensional data can quietly misbehave here.