The Error Bar Is Curvature

A likelihood curve can look like a hill.

That picture is useful, but slightly backwards. For estimation, the important part is not the hilltop. It is how fast the hill falls away from the top.

Flat hill:

many nearby parameter values explain the data about equally well

Sharp hill:

nearby parameter values stop explaining the data quickly

Fisher information is the local spring constant of that second picture. The Cramer-Rao bound says that, under the usual regularity conditions, an unbiased estimator cannot have variance smaller than the reciprocal of that information.

So the slogan is:

curvature buys precision

But only locally, only under the model, and only with the assumptions visible.

The Score Is the Local Witness

Let \(X \sim p_\theta(x)\) and write the log-likelihood for one observation as:

\[\ell(\theta;X) = \log p_\theta(X).\]

The score is the derivative:

\[S_\theta(X) = \frac{\partial}{\partial \theta}\ell(\theta;X).\]

When differentiation and integration can be exchanged, the score has mean zero:

\[\mathbb{E}_\theta[S_\theta(X)] = 0.\]

Fisher information is the score variance:

\[I(\theta) = \mathbb{E}_\theta[S_\theta(X)^2].\]

Equivalently, in regular cases, it is the negative expected second derivative of the log-likelihood:

\[I(\theta) = -\mathbb{E}_\theta \left[ \frac{\partial^2}{\partial\theta^2} \ell(\theta;X) \right].\]

The first form says information is how violently the log-likelihood changes under the true parameter. The second says information is expected curvature.

R. A. Fisher’s 1922 paper made maximum likelihood and information central tools for theoretical statistics.¹ C. R. Rao’s 1945 paper gave the inequality that now carries both Cramer and Rao’s names in much of the literature.² The modern name also honors Harald Cramer, whose 1946 text developed the same lower-bound idea in the asymptotic estimation program.³

The history matters less here than the mechanism. The bound is not magic. It is Cauchy-Schwarz.

The Bound Is a Covariance Argument

Suppose \(T(X)\) is an unbiased estimator of \(\theta\):

\[\mathbb{E}_\theta[T(X)] = \theta.\]

Differentiate both sides with respect to \(\theta\):

\[\frac{\partial}{\partial\theta} \mathbb{E}_\theta[T(X)] = 1.\]

Under the same regularity assumptions, the derivative can move inside the integral:

\[1 = \int T(x)\frac{\partial}{\partial\theta}p_\theta(x)\,dx = \int T(x)S_\theta(x)p_\theta(x)\,dx = \mathbb{E}_\theta[T(X)S_\theta(X)].\]

Because the score has mean zero, this is a covariance:

\[1 = \mathrm{Cov}_\theta(T,S_\theta).\]

Now apply Cauchy-Schwarz:

\[1^2 \le \mathrm{Var}_\theta(T) \mathrm{Var}_\theta(S_\theta) = \mathrm{Var}_\theta(T) I(\theta).\]

Therefore:

\[\mathrm{Var}_\theta(T) \ge \frac{1}{I(\theta)}.\]

For \(n\) independent observations, information adds:

\[I_n(\theta) = n I_1(\theta),\]

so the bound shrinks like:

\[\frac{1}{I_n(\theta)}.\]

That is the clean version. The fine print is also part of the theorem: the support should not move with \(\theta\) in a troublesome way, derivatives and integrals need to commute, the estimator is assumed unbiased, and boundary cases require care. Biased estimators have a modified bound. Misspecified models need a sandwich-style correction, not blind curvature worship.

Two Models Where the Bound Is Met

The lab below uses two intentionally friendly estimation problems.

Normal mean, known variance.

If \(X_i \sim N(\mu,\sigma^2)\) and \(\sigma\) is known, then:

\[I_n(\mu) = \frac{n}{\sigma^2}.\]

The sample mean is unbiased and has:

\[\mathrm{Var}(\bar{X}) = \frac{\sigma^2}{n} = \frac{1}{I_n(\mu)}.\]

It reaches the Cramer-Rao bound exactly.

Bernoulli probability.

If \(X_i \sim \mathrm{Bernoulli}(p)\), then:

\[I_n(p) = \frac{n}{p(1-p)}.\]

The sample proportion is unbiased and has:

\[\mathrm{Var}(\hat{p}) = \frac{p(1-p)}{n} = \frac{1}{I_n(p)}.\]

This also reaches the bound. That does not mean every problem is so polite. It means these two examples are good rulers for checking whether the lab’s curvature, score variance, and estimator variance are speaking the same language.

Model Sample size 80 Monte Carlo experiments 1600 Normal mean 0.40 Normal sigma 1.30 Bernoulli p 0.35 Random seed 17

What the Lab Measures

For the default normal-mean setting:

n = 80
sigma = 1.30
Fisher information = 80 / 1.30^2 = 47.3373
CRLB = 1 / I = 0.021125
Monte Carlo estimator variance = 0.021673
variance / bound = 1.026

The empirical variance is not exactly the bound because the lab runs a finite Monte Carlo experiment. The theoretical variance of the sample mean is exactly the bound. The simulation is there to check the mechanism, not to prove the theorem by dice.

The score panel checks the companion identity:

\[\mathrm{Var}_\theta(S_\theta) = I(\theta).\]

In the default run, score variance divided by information is also 1.026, because for the normal mean the score is a scaled version of the sample mean.

Reproducibility Check

The browser lab is backed by a CommonJS module. The sweep reports 35 named checks: seven information, score, bias, and likelihood invariants across five deterministic Monte Carlo cases. From the repository root:

node - <<'NODE'
const lab = require("./assets/js/fisher-information-lab.js");
const EXPECTED_CASE_SHAPE = [
  "1:normal:n=80:experiments=1600:seed=17:47.3373:0.021125:0.021673:1.026:1.026:-2.031e-3:7/7",
  "2:normal:n=25:experiments=1800:seed=8:5.1653:0.193600:0.198498:1.025:1.025:3.398e-3:7/7",
  "3:normal:n=220:experiments=1200:seed=99:448.9796:0.002227:0.002288:1.027:1.027:7.258e-4:7/7",
  "4:bernoulli:n=60:experiments=2200:seed=23:263.7363:0.003792:0.003795:1.001:1.001:1.364e-4:7/7",
  "5:bernoulli:n=180:experiments=1800:seed=77:892.8571:0.001120:0.001037:0.926:0.926:-3.241e-4:7/7"
];
const EXPECTED_CHECK_NAMES = [
  "Fisher information is positive",
  "CRLB is reciprocal information",
  "Monte Carlo variance is near CRLB",
  "score variance is near information",
  "score mean is small",
  "bias is small relative to standard error",
  "likelihood peaks at true parameter"
];
const EXPECTED_CASES = EXPECTED_CASE_SHAPE.length;
const EXPECTED_CHECKS = EXPECTED_CASES * EXPECTED_CHECK_NAMES.length;
const EXPECTED_CHECK_ROWS = Array.from({ length: EXPECTED_CASES }, (_, index) =>
  `${index + 1}:` + EXPECTED_CHECK_NAMES.map((name) => `${name}=true`).join("|")
);

function sameList(actual, expected) {
  return actual.length === expected.length &&
    actual.every((value, index) => value === expected[index]);
}

function caseShape(row) {
  return `${row.caseId}:${row.model}:n=${row.sampleSize}:` +
    `experiments=${row.experiments}:seed=${row.seed}:` +
    `${row.fisher.toFixed(4)}:${row.crlb.toFixed(6)}:` +
    `${row.estimateVariance.toFixed(6)}:${row.varianceRatio.toFixed(3)}:` +
    `${row.scoreRatio.toFixed(3)}:${row.bias.toExponential(3)}:` +
    `${row.passedChecks}/${row.totalChecks}`;
}

function checkRow(row) {
  return `${row.caseId}:` +
    row.checks.map((check) => `${check.name}=${check.ok}`).join("|");
}

function rowDrifts(actual, expected) {
  return {
    missing: expected.filter((row) => !actual.includes(row)),
    extra: actual.filter((row) => !expected.includes(row))
  };
}

function hasRowDrift(drift) {
  return drift.missing.length > 0 || drift.extra.length > 0;
}

const audit = lab.auditGrid();
const failed = audit.cases.filter(
  (row) => !row.passed || row.passedChecks !== row.totalChecks
);
const shape = {
  cases: audit.cases.map(caseShape),
  checkRows: audit.cases.map(checkRow)
};
const checkRowDrifts = rowDrifts(shape.checkRows, EXPECTED_CHECK_ROWS);
const shapeErrors = [];
if (!sameList(shape.cases, EXPECTED_CASE_SHAPE)) {
  shapeErrors.push({ name: "cases", actual: shape.cases, expected: EXPECTED_CASE_SHAPE });
}
audit.cases.forEach((row) => {
  const names = row.checks.map((check) => check.name);
  if (!sameList(names, EXPECTED_CHECK_NAMES)) {
    shapeErrors.push({
      name: `case ${row.caseId} check names`,
      actual: names,
      expected: EXPECTED_CHECK_NAMES
    });
  }
});
if (hasRowDrift(checkRowDrifts)) {
  shapeErrors.push({ name: "checkRows", drift: checkRowDrifts });
}
const summary =
  `${audit.passed}/${audit.total} cases and ${audit.passedChecks}/${audit.totalChecks} checks passed`;
if (
  shapeErrors.length ||
  failed.length ||
  !audit.ok ||
  audit.total !== EXPECTED_CASES ||
  audit.totalChecks !== EXPECTED_CHECKS ||
  audit.passed !== audit.total ||
  audit.passedChecks !== audit.totalChecks
) {
  throw new Error(JSON.stringify({
    summary,
    shapeErrors,
    shape,
    checkRowDrifts,
    failed,
    errors: audit.errors,
    totals: {
      total: audit.total,
      passed: audit.passed,
      passedChecks: audit.passedChecks,
      totalChecks: audit.totalChecks
    }
  }, null, 2));
}
console.table(audit.cases.map((row) => ({
  model: row.model,
  n: row.sampleSize,
  fisher: row.fisher.toFixed(4),
  crlb: row.crlb.toFixed(6),
  empiricalVariance: row.estimateVariance.toFixed(6),
  varianceRatio: row.varianceRatio.toFixed(3),
  scoreRatio: row.scoreRatio.toFixed(3),
  checks: `${row.passedChecks}/${row.totalChecks}`,
  passed: row.passed
})));
console.log(summary);
NODE

The current audit cases are:

Model	n	Fisher I	CRLB	Empirical variance	Var / bound	Score var / I
normal	80	47.3373	0.021125	0.021673	1.026	1.026
normal	25	5.1653	0.193600	0.198498	1.025	1.025
normal	220	448.9796	0.002227	0.002288	1.027	1.027
bernoulli	60	263.7363	0.003792	0.003795	1.001	1.001
bernoulli	180	892.8571	0.001120	0.001037	0.926	0.926

Each row is deterministic for its seed. The audit allows Monte Carlo error but requires positive information, reciprocal CRLB, small bias, score mean near zero, score variance near information, and estimator variance near the bound.

What Curvature Does Not Promise

Curvature is local. It tells you what happens near the parameter value under the model. It does not promise that the likelihood is globally single-peaked, that the model is correct, that the estimator is unbiased, or that finite-sample approximations behave well near boundaries.

It is also coordinate-sensitive in the ordinary scalar form. Reparameterize from \(p\) to \(\phi=\log(p/(1-p))\) and the numeric information changes by the Jacobian. The meaningful object is the way information transforms with the parameter. This is the same reason Fisher information becomes a metric tensor in information geometry and natural-gradient methods.

So a better operational reading is:

information is local distinguishability under a declared model

If two nearby parameters generate almost the same distribution, the likelihood is flat and no unbiased estimator can reliably separate them. If nearby parameters generate noticeably different distributions, the likelihood has curvature and precision becomes affordable.

The error bar is not painted on after the estimate. Under the model, it was already hiding in the curvature.

R. A. Fisher, “On the Mathematical Foundations of Theoretical Statistics”, Philosophical Transactions of the Royal Society of London. Series A 222, 1922. Fisher develops likelihood, efficiency, and information as central concepts in estimation theory. ↩
C. R. Rao, “Information and the Accuracy Attainable in the Estimation of Statistical Parameters”, Bulletin of the Calcutta Mathematical Society 37, 1945; reprinted in Breakthroughs in Statistics, 1992. Rao states the information inequality and related efficiency conditions. ↩
Harald Cramer, Mathematical Methods of Statistics, Princeton University Press, 1946. See the Internet Archive record for bibliographic access. The book is one source of the Cramer-Rao naming tradition and develops lower bounds for statistical estimation. ↩