Author(s): Dr Swarneendu AI

Originally published on Towards AI.

Every AI Model Trains on a Curved Surface. Gradient Descent Pretends It’s Flat. Amari Proved the Cost in 1998.

You have been visualizing the loss landscape wrong. Not slightly wrong. Geometrically wrong. The space where neural networks learn is not Euclidean. It never was. And the optimizer you are using is making decisions as if it is.

After the lead, the article argues that neural-network training should be viewed through information geometry: parameter space is a curved statistical (Riemannian) manifold, where distances are defined by the Fisher information metric rather than Euclidean distance. It explains how Fisher information (and its relationship to the Hessian of KL divergence) determines the true local geometry of how probability distributions change, showing with examples that ordinary gradient descent moves in the wrong “direction” because it assumes flatness. It then connects this geometry mismatch to practical issues like plateaus and slow progress, and to failures of “optimizer correctness” under reparameterization—while natural gradient remains invariant, SGD does not. Finally, it surveys how modern practice approximates natural gradient implicitly (e.g., Adam as a diagonal Fisher approximation, K-FAC as a structured Fisher approximation, and natural-policy-gradient methods in RL/TRPO/PPO), and extends the idea to latent-space interpolation—where Euclidean straight lines can yield blurry results because they ignore manifold curvature. The piece closes by claiming the industry has largely papered over geometry with more compute/data/parameters, and questions what future optimization at frontier scale might need if these approximations become increasingly inaccurate.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI