Some optimizers ask the function for a derivative.

CMA-ES asks a colder question: which sampled points survived?

That sounds primitive until the objective is rotated, scaled, noisy, discontinuous, or hidden behind a simulator. Then a derivative-free method with a memory for successful directions starts to look less like a genetic metaphor and more like a small statistical instrument.

Thesis: the key object in covariance matrix adaptation is not the individual mutation. It is the search distribution. CMA-ES learns a coordinate system by increasing probability mass in directions that repeatedly produced better samples.

Nikolaus Hansen’s tutorial introduces CMA-ES as a randomized method for continuous, nonlinear, non-convex black-box optimization.1 The classic Hansen-Ostermeier paper put derandomization and cumulation at the center of evolution strategies: do not let each lucky mutation vanish; accumulate the directional evidence it provides.2

The official CMA-ES notes are blunt about the use case. If BFGS or conjugate gradient works, they are usually faster. CMA-ES is attractive when gradients are unavailable, unreliable, or not even meaningful: rugged landscapes, sharp bends, noise, local optima, discontinuities, outliers.3 It is a second-order flavored method without asking for first derivatives.

That distinction matters. “Evolution strategy” can sound like a population of little agents trying random stunts. The more useful picture is:

sample points from a distribution
rank them by observed objective value
move and reshape the distribution toward the winners
repeat

Natural Evolution Strategies make this distributional view explicit: optimize the parameters of a search distribution using a natural-gradient idea.4 CMA-ES arrived from a different lineage, but the family resemblance is strong. The optimizer is not merely trying points. It is learning where future tries should come from.

A Rotated Bowl Is Already a Trap

Consider the quadratic:

\[f(x) = z_1^2 + \kappa z_2^2,\]

where (z) is a rotated version of (x). The optimum is still at the origin. The function is smooth, convex, and almost embarrassingly simple.

But if (\kappa) is large, the landscape is a narrow tilted valley. A spherical search distribution wastes most of its samples across the steep wall. It must shrink its step size to avoid overshooting, and then it crawls along the flat axis.

A full covariance matrix can do something different. It can become long in the flat direction and short in the steep direction. In a quadratic basin, that is why people say CMA-ES learns something like an inverse Hessian. The covariance is not the Hessian, but its eigensystem becomes a practical coordinate system for sampling.

The algorithm below is a compact two-dimensional implementation:

  • weighted recombination from the top half of each sampled population;
  • cumulative step-size adaptation;
  • rank-one and rank-(\mu) covariance updates;
  • no restarts, active covariance update, constraints, integer variables, surrogate modeling, or noisy-ranking machinery.

So treat it as a microscope, not as a replacement for a production library.

CMA-ES mean, samples, covariance Spherical evolution strategy True flat valley axis Best sampled point

Deterministic two-dimensional experiment. The audit checks finite positive covariance matrices, monotone best-so-far ledgers, default CMA advantage over the spherical baseline, and covariance-axis alignment across a small grid of rotations, condition numbers, and seeds.

With the default settings, the lab starts both optimizers at the same point with the same population size. The objective has a 120x condition number and a 42 deg rotation. After 32 generations:

CMA-ES best value:       4.15e-10
spherical ES best value: 7.81e-3
learned axis error:      0.14 degrees
learned covariance cond: 103.5x

The exact numbers belong to this seeded toy, not to all optimization problems. The shape is the point. The spherical baseline can reduce its step size, but it cannot rotate its sampling distribution. CMA-ES can keep sampling broadly along the valley while becoming stingy across it.

Move Valley rotation to zero. The spherical baseline improves because the problem is now aligned with the coordinate axes. Move Condition number upward. The cost of using the wrong coordinates rises. Reduce Generations to 8 or

  1. Adaptation is not free: the optimizer first has to spend samples learning the metric it wishes it had started with.

The exported lab API also runs a small verification grid. For rotations 0 deg, 25 deg, and 50 deg, condition numbers 30x, 120x, and 300x, and two seeds, all 18 cases produce finite positive covariance matrices, monotone best-so-far histories, and strong CMA improvement from the initial point.

Ranking Is the Interface

CMA-ES does not need the actual scale of the objective in the way gradient descent does. In its basic form, it sorts candidate points and uses the ranks. If one sampled point is better than another, it gets more influence. If a monotone transformation changes all objective values while preserving the order, the ranking information is the same.

That is a quiet superpower. Simulator objectives are often arbitrary mixtures of penalties:

\[\text{loss} = 10^4 \cdot \text{crash} + 7.2 \cdot \text{energy} + 0.03 \cdot \text{latency} + \text{soft constraint}.\]

The absolute numbers may be ugly. The ordering may still be meaningful enough to learn from.

But rank-only information is also a limitation. If the objective is flat across most samples and cliffs near failure, ranking can be poor evidence. If the evaluation noise is larger than the differences between candidates, the optimizer adapts to noise. If constraints hide all interesting feasible points, the search distribution can confidently learn the wrong region.

Derivative-free does not mean assumption-free. It means the assumptions have changed: sample budget, ranking reliability, variable scaling, restart policy, and constraint handling become first-class modeling choices.

The Covariance Is a Memory

At generation (t), write the search distribution as roughly

\[x \sim m_t + \sigma_t \mathcal{N}(0, C_t).\]

The mean (m_t) says where the optimizer currently believes good points live. The scalar (\sigma_t) says how far it is willing to jump. The covariance matrix (C_t) says which directions deserve larger or smaller mutations.

The best (\mu) samples form a weighted displacement:

\[y_w = \sum_{i=1}^{\mu} w_i \frac{x_{i:\lambda} - m_t}{\sigma_t}.\]

The lab updates the mean by this weighted step, accumulates an evolution path, and updates the covariance using a rank-one path term plus a rank-(\mu) sample covariance term. The production algorithm has many careful details, but the statistical sentence is compact:

directions that repeatedly contain selected points become easier to sample

This is why the covariance ellipse in the lab slowly turns into the valley. Individual samples are noisy. A path of selected samples is a memory.

Why Not Always Use It?

Because function evaluations are expensive.

On a clean convex quadratic with reliable gradients, quasi-Newton methods should usually win. The CMA-ES documentation says this plainly: second-order derivative-based methods are usually faster when they are successful, and CMA-ES is aimed at harder black-box settings.3

The more interesting decision is not “CMA-ES or gradient descent?” It is:

  • Can I compute a trustworthy derivative?
  • Is the objective smooth enough for that derivative to mean what I need?
  • Is the evaluation budget closer to hundreds, thousands, or millions?
  • Are variables scaled so a shared initial step size is sane?
  • Do constraints, simulator failures, or noise dominate the ranking?
  • Should I use restarts or a larger population for multimodal search?

Salimans, Ho, Chen, Sidor, and Sutskever revived evolution strategies in a modern reinforcement-learning context partly because ES communicates only scalar returns and parallelizes easily.5 That is a different regime from small-dimensional CMA-ES, but the shared moral is useful: sometimes the expensive thing is not taking a derivative. Sometimes it is coordinating the experiment.

A Coordinate System You Earned

The lab’s rotated bowl is deliberately too simple. It is a picture of one mechanism, not a benchmark claim.

Still, the picture contains the lesson I want to remember. A black-box optimizer is not blind just because it lacks gradients. It can infer geometry from selection. It can learn that one direction is safe to explore and another is a wall. It can carry that memory forward as a covariance matrix.

The search distribution is the optimizer’s notebook.

  1. Nikolaus Hansen, “The CMA Evolution Strategy: A Tutorial,” arXiv:1604.00772, 2016; revised 2023. arXiv. ↩

  2. Nikolaus Hansen and Andreas Ostermeier, “Completely Derandomized Self-Adaptation in Evolution Strategies,” Evolutionary Computation 9(2), 159-195, 2001. PDF, MIT Press. ↩

  3. The CMA-ES project page describes CMA-ES as a method for difficult nonlinear, non-convex black-box optimization and notes both the inverse-Hessian analogy and the cases where derivative-based methods are usually faster. CMA-ES project. The bundled c-cmaes documentation gives similar practical guidance. c-cmaes doc. ↩ ↩2

  4. Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Juergen Schmidhuber, “Natural Evolution Strategies,” Journal of Machine Learning Research 15, 949-980, 2014. JMLR, PDF. ↩

  5. Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever, “Evolution Strategies as a Scalable Alternative to Reinforcement Learning,” arXiv:1703.03864, 2017. arXiv. ↩