When we shipped policy dispatch, the first instinct from every pilot team was the same: turn every policy on, point it at every agent, and let the platform fire whenever it can. That instinct is exactly backwards. An always-fire policy does not buy you safety — it buys you alert fatigue, run-away remediation runs, and a defense layer that pages your on-call for things that were never broke enough to need a fix. Every policy needs a threshold gate, and the gate needs to be deliberate.
FleetPulse now ships that gate on every RemediationPolicy. Below is a walkthrough of what the gate does, the actual code that runs on every dispatch, and how to read the threshold number you should set for the agents you ship.
The gate
What the threshold actually does
A RemediationPolicy can now declare an errorRate threshold. Before the dispatch path runs, FleetPulse reads the latest AgentMetric for the affected agent and compares its errorRate against that threshold. If the agent is below the threshold, the policy is skipped — logged as skipped in the dispatch summary, no remediation action taken, no audit entry, no Slack ping. The agent keeps running; nobody gets paged.
A null threshold falls back to the legacy always-fire behavior, so existing policies keep their semantics. Setting a numeric threshold turns the policy into a gate: the agent has to earn a remediation by being degraded enough to matter.
// src/lib/business/agent-health.ts — runPolicyDispatch()
if (policy.threshold != null) {
if (metric.errorRate == null || metric.errorRate < policy.threshold) {
skipped += 1;
items.push({ agentId: policy.agentId, status: 'skipped' });
continue;
}
}That is the entire gate. Two lines short-circuit every always-fire policy: a missing metric skips (no data, no dispatch), and an errorRate that has not yet climbed past the threshold skips (below the line, no dispatch). Anything else falls through to the existing dispatch path — cooldown check, action dispatch, audit log.
- false-y threshold ⇒ skip the gate ⇒ resume with the dispatch path (legacy always-fire).
- threshold set, metric missing ⇒ skip the gate (logged as skipped).
- threshold set, metric present, errorRate below threshold ⇒ skip.
- threshold set, errorRate above threshold ⇒ proceed to dispatch + cooldown + audit.
Why
Why always-fire policies create alert fatigue
Most AI workloads oscillate. Error rate climbs to 2% for ten minutes during a bad batch window, then settles back to 0.4%. Latency doubles for a single run because the prompt was huge. An agent restarts because of a deploy and reports an error on its first retry. None of those events are worth a remediation — but under an always-fire policy, each one is.
The consequence shows up on the on-call rotation within a week. Every Slack notification looks the same as the ones that mattered, so nobody threads real incidents above the noise. The first time remediation actually misfires — because it will, eventually — nobody notices, because every other notification was already being muted. The system becomes a pager that you ignore, which is the worst place for an automated defense layer to land.
A threshold gate flips the bias. Instead of “dispatch by default, suppress on explicit opt-out,” the policy becomes “dispatch only when the signal is strong enough,” and the question — what counts as strong enough — becomes a deliberate decision that lives on the policy itself.
Calibration
Reading the threshold number that matters to you
The right threshold is not a universal constant. It depends on what “broken” means for the agent in question — a customer-facing support bot can tolerate 1% errors for a minute, but a billing agent cannot tolerate 0.1% errors ever. Start from the agent’s SLO: a sensible threshold is the error rate at which the agent has already violated its reliability target, minus a little headroom so the gate fires before the SLO trips, not after.
In practice, most fleets ship with two bands. A low band (say 0.5% to 2%) for low-stakes agents where remediation is cheap and noisy fires are tolerable. A high band (0.05% and below) for high-stakes agents where any false-positive remediation is more damaging than the failure it would prevent. Tune per agent, not per fleet.
- Start from the agent’s SLO, not from another agent’s number.
- Set the gate BELOW the SLO so remediation precedes the breach, not follows it.
- Re-tune monthly — traffic shape changes and yesterday’s right threshold is today’s false positive.
- Watch the dispatched vs. skipped count in the dispatch summary; if skipped >> dispatched, the gate is doing its job.