01 - One-Hot Is a Choice

One-Hot Is a Choice

The default encoding of class labels is not a neutral container. It is a specific geometry, and it makes a specific claim.

The routine decision

Take a text classification problem: $n$ classes, one encoder, one head. You encode each class as a one-hot vector: a $1$ in position $i$, zeros elsewhere. Then you train against it. The decision is sensible, it is what everyone does, and it is almost never described as a decision.

But it is a map. It sends the label set $\{1,\dots ,n\}$ into ${\mathbb{R}}^n$, and the image is a particular configuration of points. Worth asking which one.

One-hot, centred, is a regular simplex

Write $e_i$ for the $i$-th standard basis vector and centre the set over its own mean:

$${\tilde{e}}_i=e_i-\frac{1}{n}1.$$

The inner products come out immediately:

$$⟨{\tilde{e}}_i,{\tilde{e}}_j⟩={\delta }_{ij}-\frac{1}{n}.$$

So the Gram matrix has $(n-1)/n$ everywhere on the diagonal and $-1/n$ everywhere off it. Two consequences, and they are the whole point:

Every label sits at the same distance from the centroid. The diagonal is constant, so $\Vert {\tilde{e}}_i{\Vert }^2=(n-1)/n$ for all $i$. No class is more central than any other.

Every pair of labels sits at the same distance from every other pair. For $i\ne j$,

$$\Vert {\tilde{e}}_i-{\tilde{e}}_j{\Vert }^2=\frac{n-1}{n}+\frac{n-1}{n}+\frac{2}{n}=2.$$

Constant, independent of which pair. That configuration is the regular simplex: $n$ points, all pairwise distances equal. It lives in the $(n-1)$-dimensional hyperplane $\{x:1^{\top }x=\text{const}\}$.

Info

Centring is a change of origin, not a change of geometry. A decoder that scores a prediction by its distance to each class point sees only differences between class points, and differences are invariant to translation. So the one-hot code and its centred version are the same object as far as the decision rule is concerned.

The extra coordinate is the familiar bookkeeping oddity: one-hot has $n$ columns while the simplex it defines has rank $n-1$. The component along $1$ is common to all $n$ points and cancels in every difference.

What equidistance asserts

Equal distances are not an absence of a claim. They are a claim: no pair of classes is a priori more confusable than any other pair.

A topic corpus makes the cost of that claim concrete. The 20 Newsgroups collection (Lang, 1995) is a standard text-classification benchmark. Its twenty classes are Usenet discussion groups, topic-based forums that predate the web, and its labels are the group names themselves. Take three of them:

comp.sys.ibm.pc.hardware
comp.sys.mac.hardware
soc.religion.christian

The first two are near neighbours in any reasonable sense, and the corpus says so out loud: they share the prefix comp.sys, because Usenet names are dotted paths and the naming scheme is a declared hierarchy. The third sits under a different top-level branch entirely, and is far from both. The default target says all three are at distance $\sqrt{2}$ from one another.

Three 20 Newsgroups labels under two geometries, abbreviated here to their distinguishing tails: in full they are comp.sys.ibm.pc.hardware, comp.sys.mac.hardware and soc.religion.christian. In (a) the dashed circle is the set of points equidistant from the centroid, and all three classes sit on it. The centred one-hot code has equal radii and equal pairwise distances, so no pair is closer than any other, including the two that share the comp.sys prefix. In (b) the two hardware groups share a region and the religion group is far from both. Schematic: positions are drawn, not measured.

That is not obviously wrong. Confusing the two hardware groups may cost exactly as much as confusing either with religion. That depends on the application, not on the corpus. But it is a position, it was taken by default, and a position taken by default is worth at least identifying as one.

The same shape, arrived at from three directions

What makes the simplex interesting is that it does not only appear as the accident of a convenient encoding. Three separate literatures converge on it, for reasons that have nothing to do with each other.

Literature	How the simplex enters
Neural collapse	The terminal phase of training drives class means to a simplex equiangular tight frame, an emergent attractor rather than a design (Papyan, Han & Donoho, 2020)
Frozen classifier heads	The last layer is fixed to an equiangular frame in advance, and training the head turns out to be largely unnecessary (Hoffer, Hubara & Soudry, 2018; Yang et al., 2022)
Hyperspherical prototypes	Class prototypes are spread on a sphere by maximising minimum separation, which in the unconstrained case is the same configuration (Mettes, van der Pol & Snoek, 2019)

One community finds it at the end of optimisation, one puts it in at the start, one derives it from a packing argument. And the one-hot target has been sitting there the whole time, being it by construction. The relation between class encoding and the collapse geometry is itself an active question (Massion, Makhlouf & Massart, 2026).

If it is a map, why this map

Nothing forces the equidistance. You can use fewer than $n-1$ dimensions. You can place classes at unequal distances, so that labels the decoder already confuses share a region of the space at no extra cost. Several standard designs already do exactly this:

• Zero-shot and label-embedding classifiers embed the class names with a text encoder and use those embeddings as the output space (Palatucci et al., 2009; Akata et al., 2015).
• Error-correcting output codes assign each class a binary codeword, deliberately far apart in Hamming distance (Dietterich & Bakiri, 1995).
• Hierarchical methods inject a declared taxonomy so that errors within a branch cost less than errors across branches (Bertinetto et al., 2020).

Each of these puts information into the labels before any training happens. The natural question is what that information buys, and the natural obstacle is that there is no unit in which to answer.

Before the unit, though, there is a prior problem: the information going into a label space comes in two kinds, and they are not the same kind. That is the next article.

---

References

• Akata, Z., Perronnin, F., Harchaoui, Z., & Schmid, C. (2016). Label-Embedding for Image Classification. IEEE TPAMI, 38(7), 1425–1438.
• Bertinetto, L., Mueller, R., Tertikas, K., Samangooei, S., & Lord, N. A. (2020). Making Better Mistakes: Leveraging Class Hierarchies with Deep Networks. CVPR.
• Dietterich, T. G., & Bakiri, G. (1995). Solving Multiclass Learning Problems via Error-Correcting Output Codes. JAIR, 2, 263–286.
• Hoffer, E., Hubara, I., & Soudry, D. (2018). Fix your classifier: the marginal value of training the last weight layer. ICLR.
• Lang, K. (1995). NewsWeeder: Learning to Filter Netnews. Machine Learning Proceedings 1995, 331–339.
• Massion, B., Makhlouf, R., & Massart, E. (2026). The role of class encoding in neural collapse. arXiv:2606.00344.
• Mettes, P., van der Pol, E., & Snoek, C. G. M. (2019). Hyperspherical Prototype Networks. NeurIPS.
• Palatucci, M., Pomerleau, D., Hinton, G., & Mitchell, T. (2009). Zero-shot Learning with Semantic Output Codes. NeurIPS.
• Papyan, V., Han, X. Y., & Donoho, D. L. (2020). Prevalence of neural collapse during the terminal phase of deep learning training. PNAS, 117(40), 24652–24663.
• Yang, Y., Chen, S., Li, X., Xie, L., Lin, Z., & Tao, D. (2022). Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?. NeurIPS.