fin1te

Writing/·3 min read

The 497-day bug

Healthy switches and routers were showing up as down in SLA reports. The common factor was uptime: about 497 days of it, which is exactly where a 32-bit counter of hundredths of a second runs out.

SNMPCiscoMonitoringSpark

SLA reports for a managed cloud platform were flagging downtime on network devices that nobody believed had gone down. No tickets, no alerts from the NOC, no users complaining. Just outages in a report that feeds contractual penalties.

The report came from the downtime engine I built: a Spark job that reads monitoring data for every switch, router and firewall in the estate and works out, window by window, whether each one was up.

The clue

The affected devices had nothing obvious in common. Different models, different sites, a mix of Cisco switches and routers. What they did share was a long uptime. One of them had been running for 497 days.

If you have worked with SNMP, that number rings a bell.

Why 497

The standard uptime object every SNMP agent exposes is sysUpTime (OID 1.3.6.1.2.1.1.3.0). It is a TimeTicks value: hundredths of a second since the agent started, stored as an unsigned 32-bit integer.

2^32 ticks / 100 ticks per second = 42,949,672.96 seconds
42,949,672.96 / 86,400             = 497.1 days

After that the counter wraps back to zero. The device is fine; the number just rolls over like an odometer. Cisco even has a bug record for it with the title “sysUpTime will wrap after about 497 days, 2:27:35.80”, which is 2³² hundredths of a second written out.

It is not only sysUpTime. hrSystemUptime from HOST-RESOURCES-MIB has the same type and wraps at the same point. On some older IOS releases, snmpEngineTime, the counter people often switch to as a workaround, was found to reset at the same moment sysUpTime wrapped, which is its own Cisco bug.

The number turns up outside networking too. On 32-bit Linux, the kernel’s jiffies counter at HZ=100 wrapped after the same 497 days, and early 2000s kernels showed uptime starting again from zero until patches exported a 64-bit value. At HZ=1000 it wrapped after 49.7 days, the same figure behind the old Windows 95 and 98 hang, caused by a 32-bit millisecond timer.

Why the parser believed it

A common way to detect a reboot is “uptime went down since the last sample”. That rule cannot tell a restart from a wrap. To the parser, a device whose counter wrapped looked exactly like one that had just rebooted, and a reboot implies it was down for a while before that.

The fix

The engine now treats an uptime decrease as a question rather than an answer:

  1. Is it a wrap? If the previous sample was close to the 32-bit ceiling and the new value is consistent with the wall-clock time that has passed, the counter wrapped. No reboot happened.
  2. Was the device reachable? Before recording downtime, check the ICMP reachability samples for the same window. A device that answered throughout was not down, whatever its counters say.

Monitoring stores uptime in seconds, so the ceiling is 42,949,672. In SQL the guard is a single extra condition on the restart query:

AND (
  prev_value <= 42949000                      -- far from the ceiling: a real restart
  OR NOT EXISTS (                             -- near it: only count if ping failed too
    SELECT 1 FROM history_uint ping
    WHERE ping.itemid = :icmp_item_for_host
      AND ping.clock >  FLOOR(prev_clock / 60) * 60
      AND ping.clock <= CEILING((clock - value) / 60) * 60
      AND ping.value = 1                      -- host answered: wrap, not downtime
  )
)

A second, smaller issue turned up on the way: the ping window itself. Ping samples land on minute boundaries, so the window has to be widened outward, the start floored to the minute and the end ceilinged. Before that alignment, a sample sitting exactly on a boundary could fall outside the window, and a healthy device looked unreachable for exactly the moment that mattered.

Together they removed the false outages without hiding real ones.

The same trap, in traffic counters

Uptime is the famous case, but interface counters have the same problem and much faster. The original IF-MIB octet counters, ifInOctets and ifOutOctets, are 32-bit Counter32 values:

2^32 bytes × 8                 = 34.4 gigabits
at 1 Gbps:  34.4 Gb / 1 Gb/s  ≈ 34 seconds to wrap
at 10 Gbps:                   ≈ 3.4 seconds
at 100 Gbps:                  ≈ 0.34 seconds

Poll a 10G link every five minutes and a 32-bit counter will have wrapped dozens of times between samples, so the difference you compute is meaningless. That is why the 64-bit “high capacity” counters (ifHCInOctets, ifHCOutOctets) exist, and why any monitoring job that still reads the 32-bit versions on fast links is quietly wrong.

Testing it without touching production

A report that drives penalties is not something to experiment on. I set up parallel, isolated runs of the downtime jobs so the new logic could be compared against the old on the same inputs, and only promoted it once the only differences were the false outages it was meant to remove.

Takeaways

  • If a problem only affects long-running things, look for a counter that overflows. 497 days (32-bit hundredths of a second) and 49.7 days (32-bit milliseconds) are the two classics.
  • Prefer counters that cannot wrap in any realistic lifetime: snmpEngineTime for uptime where the agent implements it properly, and the 64-bit HC counters for traffic.
  • Never infer an event from one derived signal when you have a direct one. Uptime is an inference about reachability; reachability data is the thing itself.
  • Anything that feeds money or contracts deserves a shadow run before release.