A heartbeat is not a proof of life.

It is a sample from a path that was recently alive enough to deliver a message. When the samples stop, the observer has not learned “the process crashed.” It has learned a smaller fact:

no heartbeat has arrived for t milliseconds

Everything else is policy.

That distinction matters because asynchronous distributed systems have no local observation that cleanly separates a crashed process from a slow process. Chandra and Toueg’s failure-detector model was built around that discomfort: the detector can be unreliable, and the protocol using it has to say what kind of mistakes it can tolerate.1

The fixed-timeout version hides the policy inside a single number:

if silence exceeds 3 seconds, mark the peer down

That is often serviceable. It is also blunt. A three-second gap means something different on a quiet LAN, a noisy mobile link, a JVM during a stop-the-world pause, and a cloud VM under bursty scheduling delay. The same clock reading can be ordinary in one history and alarming in another.

The phi-accrual failure detector changes the shape of the question:

how surprising is this much silence, given the recent heartbeat history?

Accrue Suspicion, Not Verdicts

Hayashibara, Defago, Yared, and Katayama introduced the phi-accrual detector as an “accrual” detector: the monitoring component outputs a suspicion level instead of a Boolean alive/dead answer.2 Applications then choose their own thresholds. A cluster membership service, a speculative read path, and a leader-election path do not need to attach the same cost to the same silence.

The usual score is:

\[\phi(t) = -\log_{10}(P_\text{later}(t)) = -\log_{10}(1 - F(t)),\]

where \(F\) is an estimated cumulative distribution function for heartbeat inter-arrival times. In the original paper, the detector keeps a sliding window of recent heartbeat intervals and models those intervals with a normal distribution.2

That logarithm is a useful engineering unit. If the model says a heartbeat arriving this late or later has probability 0.001, then:

\[\phi = -\log_{10}(0.001) = 3.\]

So increasing the threshold by one decimal order asks for roughly ten times stronger evidence from silence. That is not magic calibration. It is calibration relative to the detector’s model, window, and workload.

Akka’s documentation describes the same shape for its phi accrual failure detector: compute phi from the normal CDF of historical heartbeat inter-arrival times, return the phi value, and let the consumer interpret it.3 It also adds an operationally important knob, an acceptable heartbeat pause, because production systems see garbage collection pauses, scheduling hiccups, and network stalls that are not peer crashes.

Lab: Score The Silence

The lab below simulates one observer monitoring one peer. The peer sends heartbeats with a nominal period, random jitter, and occasional long pauses. At 26 s the peer crashes and no more heartbeats arrive.

Two policies listen to the same trace:

fixed timeout: suspect when elapsed silence exceeds a fixed millisecond budget
phi accrual:   estimate the recent interval distribution and threshold phi

The implementation is intentionally small. It uses a deterministic random seed, a bounded sliding window of inter-arrival samples, a normal approximation for the heartbeat CDF, and a configurable pause margin:

effective_elapsed = max(0, elapsed_since_last_heartbeat - pause_margin)
tail = 1 - normal_cdf(effective_elapsed, recent_mean, recent_stddev)
phi = -log10(tail)

That margin is not the essence of phi accrual. It is a production-flavored acknowledgment that the monitor should not confuse every local scheduling pause with remote death.

phi score and threshold fixed timeout / long pauses ordinary heartbeat arrivals crash and false-suspicion bands

The lab is deterministic for a given seed. It is not a full membership protocol: there are no indirect probes, incarnation numbers, quorum rules, repair paths, or correlated multi-node failures.

The default run generates 26 heartbeat arrivals before the crash. The final window estimates a mean inter-arrival time of about 962 ms with a standard deviation of about 157 ms. With threshold 4.0 and a 250 ms pause margin, the phi detector crosses its threshold after roughly 1.80 s of silence.

On that exact trace:

Detector False-suspicion episodes before crash First suspicion after crash
Phi threshold 4.0 0 900 ms
Fixed 1200 ms timeout 1 300 ms

The fixed timeout is faster here because it is deliberately aggressive: it fires only 200 ms above the nominal heartbeat interval. It also pays for that aggression before the crash. The phi detector is not “better” in the abstract. It is making the tradeoff explicit:

lower threshold -> faster detection, more false suspicion
higher threshold -> slower detection, fewer false suspicion events

You can reproduce the audit from Node:

const lab = require("./assets/js/phi-accrual-lab.js");
const CASE_PARAMS = [
  {
    label: "default noisy path",
    params: {
      fixedTimeout: 1200,
      heartbeat: 1000,
      jitter: 140,
      pauseChance: 10,
      pauseMargin: 250,
      pauseSize: 1800,
      phiThreshold: 4,
      seed: 92
    }
  },
  {
    label: "quiet path",
    params: {
      fixedTimeout: 1600,
      heartbeat: 800,
      jitter: 60,
      pauseChance: 3,
      pauseMargin: 150,
      pauseSize: 900,
      phiThreshold: 3,
      seed: 7
    }
  },
  {
    label: "bursty path",
    params: {
      fixedTimeout: 3600,
      heartbeat: 1200,
      jitter: 260,
      pauseChance: 12,
      pauseMargin: 600,
      pauseSize: 2400,
      phiThreshold: 6,
      seed: 87
    }
  },
  {
    label: "very noisy path",
    params: {
      fixedTimeout: 5200,
      heartbeat: 1500,
      jitter: 450,
      pauseChance: 18,
      pauseMargin: 900,
      pauseSize: 3200,
      phiThreshold: 8,
      seed: 123
    }
  }
];
const EXPECTED_CASE_SHAPE = [
  "default noisy path:seed=92:heartbeat=1000:threshold=4:pauseMargin=250:phiDetect=900:fixedDetect=300:phiFalse=0:fixedFalse=1:mean=962.3:std=156.8:checks=7/7",
  "quiet path:seed=7:heartbeat=800:threshold=3:pauseMargin=150:phiDetect=1400:fixedDetect=1100:phiFalse=2:fixedFalse=2:mean=877.2:std=273.2:checks=7/7",
  "bursty path:seed=87:heartbeat=1200:threshold=6:pauseMargin=600:phiDetect=5700:fixedDetect=2600:phiFalse=1:fixedFalse=1:mean=1660.7:std=932.0:checks=7/7",
  "very noisy path:seed=123:heartbeat=1500:threshold=8:pauseMargin=900:phiDetect=3200:fixedDetect=4100:phiFalse=0:fixedFalse=0:mean=1462.9:std=339.4:checks=7/7"
];
const EXPECTED_CHECK_NAMES = [
  "arrivals are strictly increasing",
  "arrivals stop before crash",
  "phi scores are finite and nonnegative",
  "sample window is bounded",
  "phi eventually detects the crash",
  "fixed timeout eventually detects the crash",
  "higher phi threshold does not detect earlier"
];
const EXPECTED_CHECK_SHAPE = CASE_PARAMS.flatMap((entry) =>
  EXPECTED_CHECK_NAMES.map((name) => `${entry.label}:${name}:true`)
);
const EXPECTED_CASES = EXPECTED_CASE_SHAPE.length;
const EXPECTED_CHECKS = EXPECTED_CHECK_SHAPE.length;

function sameList(actual, expected) {
  return actual.length === expected.length &&
    actual.every((value, index) => value === expected[index]);
}

function caseShape(row) {
  return `${row.label}:seed=${row.seed}:heartbeat=${row.heartbeat}:` +
    `threshold=${row.phiThreshold}:pauseMargin=${row.pauseMargin}:` +
    `phiDetect=${row.phiDetectionMs}:fixedDetect=${row.fixedDetectionMs}:` +
    `phiFalse=${row.phiFalse}:fixedFalse=${row.fixedFalse}:` +
    `mean=${row.mean.toFixed(1)}:std=${row.std.toFixed(1)}:` +
    `checks=${row.passedChecks}/${row.totalChecks}`;
}

function checksFor(entry) {
  const result = lab.evaluate(entry.params);
  return [
    ["arrivals are strictly increasing", result.audit.arrivalsIncreasing],
    ["arrivals stop before crash", result.audit.arrivalsBeforeCrash],
    ["phi scores are finite and nonnegative", result.audit.finitePhi],
    ["sample window is bounded", result.audit.windowBounded],
    ["phi eventually detects the crash", result.audit.phiEventuallyDetects],
    ["fixed timeout eventually detects the crash", result.audit.fixedEventuallyDetects],
    ["higher phi threshold does not detect earlier", result.audit.thresholdMonotone]
  ].map(([name, passed]) => `${entry.label}:${name}:${passed}`);
}

const audit = lab.auditPhiAccrualLab();
const shape = {
  cases: audit.cases.map(caseShape),
  checks: CASE_PARAMS.flatMap(checksFor)
};
const shapeErrors = [];
if (!sameList(shape.cases, EXPECTED_CASE_SHAPE)) {
  shapeErrors.push({ name: "cases", actual: shape.cases, expected: EXPECTED_CASE_SHAPE });
}
if (!sameList(shape.checks, EXPECTED_CHECK_SHAPE)) {
  shapeErrors.push({
    name: "checks",
    actual: shape.checks,
    expected: EXPECTED_CHECK_SHAPE
  });
}
const failedCases = audit.cases.filter(
  (row) => !row.passed || row.passedChecks !== row.totalChecks
);
if (shapeErrors.length ||
    !audit.ok ||
    audit.cases.length !== EXPECTED_CASES ||
    audit.total !== EXPECTED_CASES ||
    audit.totalChecks !== EXPECTED_CHECKS ||
    failedCases.length ||
    audit.passed !== audit.total ||
    audit.passedChecks !== audit.totalChecks) {
  throw new Error(JSON.stringify({
    shapeErrors,
    failures: audit.failures,
    failedCases,
    totals: {
      rows: audit.cases.length,
      passed: audit.passed,
      passedChecks: audit.passedChecks,
      total: audit.total,
      totalChecks: audit.totalChecks
    }
  }, null, 2));
}

console.log(`${audit.passed}/${audit.total} audit cases and ` +
  `${audit.passedChecks}/${audit.totalChecks} checks passed`);
console.table(audit.cases.map((row) => ({
  scenario: row.label,
  heartbeat: row.heartbeat,
  phiThreshold: row.phiThreshold,
  phiDetectMs: row.phiDetectionMs,
  fixedDetectMs: row.fixedDetectionMs,
  phiFalse: row.phiFalse,
  fixedFalse: row.fixedFalse,
  mean: row.mean.toFixed(1),
  std: row.std.toFixed(1)
})));

The grid keeps the audit small but varied: a default noisy path, a quiet path, a bursty path, and a very noisy path. Together they passed 28/28 named checks: monotone heartbeat arrival times, no arrivals after the crash, bounded sample windows, finite nonnegative phi scores, eventual detection by both detectors, and monotone detection delay as the phi threshold rises.

What The Number Does Not Know

The score looks probabilistic, but the detector is not a truth oracle. It is a small statistical instrument.

First, the distributional assumption is fragile. A normal model over recent inter-arrival times is convenient, not sacred. Real delay often has heavy tails, diurnal structure, congestion episodes, garbage collection pauses, and correlated failures. If the tail model is wrong, phi = 4 does not mean “one mistake in ten thousand” in the world. It means “tail probability around 10^-4 under this detector’s current model.”

Second, the sliding window learns from recent pain. That is usually a feature: when a path becomes noisy, the detector becomes less eager to convict. It can also be a trap. If the window is filled with pathological delay, a real crash can take longer to detect because the detector has learned to expect silence.

Third, suspicion is not membership. Cassandra’s older 2.2 documentation described a phi-style phi_convict_threshold inside gossip failure detection, with lower thresholds marking nodes down more readily and higher thresholds reducing transient false failures.4 That detector output still has to feed a repair story: hinted handoff, read repair, anti-entropy repair, or a human decision. Marking a node down is not the same thing as making the data safe.

Finally, a detector threshold is a product decision wearing math clothes. A load balancer may prefer quick removal and accept some false positives. A consensus system may require a separate election timeout, quorum evidence, and term fencing. A paging system may prefer to wait, aggregate evidence, or require independent observers before waking a human.

The Useful Separation

The best part of phi accrual is not the normal CDF. It is the separation of measurement from interpretation:

monitor: here is how surprising the silence is
policy: here is what this subsystem does at that suspicion level

That separation is modest and powerful. It lets one observer feed different consumers without pretending they share the same loss function. It also forces the operational question into daylight:

what is the cost of waiting?
what is the cost of being wrong?
what delay history should count as normal?
what repair path handles an incorrect suspicion?

A missed heartbeat is evidence. A phi score is a calibrated way to write down that evidence. The failure detector still does not know whether the peer is dead. It only tells the protocol how expensive the silence has become.

The simulator source is in assets/js/phi-accrual-lab.js.

  1. Tushar Deepak Chandra and Sam Toueg, “Unreliable Failure Detectors for Reliable Distributed Systems”, Journal of the ACM 43(2):225-267, 1996. ↩

  2. Naohiro Hayashibara, Xavier Defago, Rami Yared, and Takuya Katayama, “The Phi Accrual Failure Detector”, JAIST Technical Report IS-RR-2004-010, 2004. ↩ ↩2

  3. Akka documentation, “The Phi Accrual Failure Detector”. ↩

  4. DataStax documentation for Apache Cassandra 2.2, “Failure detection and recovery”. ↩