The KL divergence is not a distance, but its second derivative is a metric.
The Kullback-Leibler divergence, the relative entropy, is described as a measure of how far apart two distributions are. This paper writes that it is not a distance, that it nevertheless is one infinitesimally, and that its asymmetry is doing work. NO NEW MATHEMATICAL THEOREM AND NO NEW PHYSICAL LAW IS CLAIMED. Kullback-Leibler (1951), Csiszar’s f-divergences, Chentsov’s theorem and the difference between forward and reverse KL in variational inference are all standard.
First, it is not a distance. For the three-point distributions (0.5, 0.3, 0.2) and (0.2, 0.5, 0.3) one direction gives 0.223805 and the other 0.193794, so the same two distributions give different values when the order is reversed. Inserting (0.34, 0.33, 0.33) in between, 0.223805 exceeds the sum 0.064083 plus 0.074746, that is 0.138829, so the triangle inequality does not hold. Over two hundred thousand random triples it failed in 50826 cases, that is 25.4 per cent. It is nevertheless non-negative and vanishes only when the two distributions coincide, so whether they are the same can be told. What cannot be told is a consistent ordering of which is further from which. One word, distance, contains two demands, deciding identity and closing a triangle, and KL meets only the first.
Second, one of the two cores. Its second derivative is a metric. Shifting only the mean in a Gaussian family gives, in closed form, the square of the shift divided by twice the variance. The Fisher metric of Paper 133 has the mean component equal to the reciprocal of the variance, so the difference from half the metric times the square of the shift is zero, an exact agreement rather than an approximation. Shifting only the width, both the value and the first derivative vanish, and the second derivative is exactly two over the variance, that is, the width component of the Fisher metric. And the third-order term matters. In the width expansion its coefficient is minus five thirds, and it changes sign with the shift. Up to second order the divergence is symmetric; from third order it is not. That a thing can fail to be a distance globally while being one infinitesimally is exactly this term’s doing. Paper 133 wrote that the Fisher metric is positive definite and unique in Chentsov’s sense; this paper writes where that metric comes from, namely the second derivative of a divergence.
Third, the other core. Every f-divergence collapses onto the same metric. Expanding the contribution from one outcome whose probability moves from a to a plus e, KL gives one over two a times e squared, reverse KL the same, chi-squared gives one over a times e squared, and squared Hellinger one over four a times e squared. All four are the same quadratic form up to a constant. Four quantities that differ entirely at global scale collapse onto the same form infinitesimally, and that is why the statement that the invariant metric is the Fisher metric and no other (Chentsov, Paper 133) can hold at all. Uniqueness is another way of saying that there was only ever one candidate. Put the other way, the quadratic coefficient does not tell which divergence one is looking at, so the reason to prefer one lies entirely outside second order.
Fourth, the asymmetry is not a defect but a division of labour. Taking an equal mixture of Gaussians at minus three and plus six, each of width one, as the target, it was approximated by a single Gaussian. Minimising the forward divergence, from target to model, gives mean 1.50 and width 4.60: it stands at the midpoint of the two modes and spreads to cover them. Minimising the reverse divergence, from model to target, gives mean minus 3.00 and width 1.00: it sits exactly on one mode. The reason lies in how the penalty falls. A heavy penalty falls where the first argument is large and the second small, so in the forward direction the model is not allowed to vanish where the target lives, and it therefore covers; in the reverse direction the model is not allowed to live where the target vanishes, and it therefore dives into one mode. Reading the asymmetry as a defect to be repaired by symmetrisation is therefore a mistake. The two directions answer different questions, and which to use is decided by what one wants; symmetrising erases the distinction. As Paper 95 said, being a convention is not the same as being a free choice.
Fifth, in thermodynamics it is the excess free energy. Writing the equilibrium distribution as the Boltzmann distribution and the equilibrium free energy through the logarithm of the partition function, for any distribution the expected energy minus the entropy term exceeds the equilibrium free energy by exactly the Boltzmann constant times the temperature times the relative entropy. So the excess free energy of a non-equilibrium distribution is its relative entropy to equilibrium. Non-negativity states that the free energy cannot fall below its equilibrium value, and that the relative entropy decreases under relaxation is one way of stating the second law (Papers 88, 136). This brings up units. Information and free energy are the same quantity in different units: whether to multiply by the Boltzmann constant times the temperature, and whether the logarithm is base two or base e, are both conventions (Paper 131). What is not a convention is the single fact of non-negativity.
IMPORTANT (scope): a census of established standard material (Layer 2). Every work cited is standard. What is not claimed: any new mathematical theorem (there is none) / any new physical law (there is none) / that the KL divergence is in error (it is not) / that the asymmetry ought to be symmetrised away (it ought not; the distinction would be erased). What is added: counting the failures of the triangle inequality, showing in closed form that the second derivative is the Fisher metric, setting the quadratic coefficients of four divergences side by side, and demonstrating the asymmetry as a division of labour. What remains: the word apartness lumps together what can be measured and what cannot; shrink to the infinitesimal and every divergence becomes the same single metric, and the differences live only where the shrinking stopped.
On the making of this work: The ideas and content of this work stem from the author’s own considerations. Assistance from an AI (a large language model) was used for structuring, English translation, and checking the algebra. Any remaining errors or misinterpretations are solely the author’s. Feedback and corrections are sincerely appreciated.
Reference
Yamagishi, Y. (2026). The KL Divergence Is Not a Distance, but Its Second Derivative Is a Metric [Paper 137]. Zenodo.
DOI: 10.5281/zenodo.22081369
ORCID: 0009-0006-8587-3450