The 497-day bug
Healthy switches and routers were showing up as down in SLA reports. The common factor was uptime: about 497 days of it, which is exactly where a 32-bit counter of hundredths of a second runs out.
SLA reports for a managed cloud platform were flagging downtime on network devices that nobody believed had gone down. There were no tickets, no NOC alerts and no user complaints, only outages in a report that feeds contractual penalties.
The report came from the downtime engine I built: a Spark job that reads monitoring data for every switch, router and firewall in the estate and works out, window by window, whether each one was up.
The clue
The affected devices had nothing obvious in common: different models, different sites, a mix of Cisco switches and routers. What they did share was a long uptime. One of them had been running for 497 days.
If you have worked with SNMP, that number rings a bell.
Why 497
The standard uptime object every SNMP agent exposes is sysUpTime (OID 1.3.6.1.2.1.1.3.0). It is a TimeTicks value: hundredths of a second since the agent started, stored as an unsigned 32-bit integer.
2^32 ticks / 100 ticks per second = 42,949,672.96 seconds
42,949,672.96 / 86,400 = 497.1 daysAfter that the counter wraps back to zero. The device is fine; the number just rolls over like an odometer. Cisco even has a bug record for it with the title “sysUpTime will wrap after about 497 days, 2:27:35.80”, which is 2³² hundredths of a second written out.
It is not only sysUpTime. hrSystemUptime from HOST-RESOURCES-MIB has the same type and wraps at the same point. On some older IOS releases, snmpEngineTime, the counter people often switch to as a workaround, was found to reset at the same moment sysUpTime wrapped, which is its own Cisco bug.
The number turns up outside networking too. On 32-bit Linux, the kernel’s jiffies counter at HZ=100 wrapped after the same 497 days, and early 2000s kernels showed uptime starting again from zero until patches exported a 64-bit value. At HZ=1000 it wrapped after 49.7 days, the same figure behind the old Windows 95 and 98 hang, caused by a 32-bit millisecond timer.
Why the parser believed it
A common way to detect a reboot is “uptime went down since the last sample”. That rule cannot tell a restart from a wrap. To the parser, a device whose counter wrapped looked exactly like one that had just rebooted, and a reboot implies it was down for a while before that.
How long “a while” is comes from the restart query itself. It books downtime from the last sample before the drop until the moment the device says it came back: the time of the new sample minus the uptime it reports. After a wrap, that moment is the wrap. Every wrap became an outage of up to one polling interval, on a device that never stopped.
If you want the wrap period for another counter, I put the arithmetic in a counter wrap calculator that runs in the browser.
The fix
The engine now treats an uptime decrease as a question rather than an answer:
- Is it a wrap? If the previous sample was close to the 32-bit ceiling and the new value is consistent with the wall-clock time that has passed, the counter wrapped. No reboot happened.
- Was the device reachable? Before recording downtime, check the ICMP reachability samples for the same window. A device that answered throughout was not down, whatever its counters say.
Monitoring stores uptime in seconds, so the ceiling is 42,949,672, and the guard treats a previous sample within 672 seconds of it, about eleven minutes, as close. In SQL it is a single extra condition on the restart query:
AND (
prev_value <= 42949000 -- far from the ceiling: a real restart
OR NOT EXISTS ( -- near it: only count if ping failed too
SELECT 1 FROM history_uint ping
WHERE ping.itemid = :icmp_item_for_host
AND ping.clock > FLOOR(prev_clock / 60) * 60
AND ping.clock <= CEILING((clock - value) / 60) * 60
AND ping.value = 1 -- host answered: wrap, not downtime
)
)A second, smaller issue turned up on the way, in the ping window itself. Ping samples land on minute boundaries, so the window has to be widened outward, the start floored to the minute and the end ceilinged. Before that alignment, a sample sitting exactly on a boundary could fall outside the window, and a healthy device looked unreachable for exactly the moment that mattered.
With both changes the false outages went away and the real ones were still counted.
The same trap, in traffic counters
Uptime is the famous case, but interface counters have the same problem and much faster. The original IF-MIB octet counters, ifInOctets and ifOutOctets, are 32-bit Counter32 values:
2^32 bytes × 8 = 34.4 gigabits
at 1 Gbps: 34.4 Gb / 1 Gb/s ≈ 34 seconds to wrap
at 10 Gbps: ≈ 3.4 seconds
at 100 Gbps: ≈ 0.34 secondsPoll a busy 10G link every five minutes and a 32-bit counter can wrap as many as 87 times between samples, so the difference you compute is meaningless. That is why the 64-bit “high capacity” counters (ifHCInOctets, ifHCOutOctets) exist: at 100 Gbps one of those takes about 47 years to wrap. Any monitoring job that still reads the 32-bit versions on fast links is quietly wrong.
Testing it without touching production
A report that drives penalties is not something to experiment on. I set up parallel, isolated runs of the downtime jobs so the new logic could be compared against the old on the same inputs, and only promoted it once the only differences were the false outages it was meant to remove.
Takeaways
- If a problem only affects long-running things, look for a counter that overflows. 497 days (32-bit hundredths of a second) and 49.7 days (32-bit milliseconds) are the two classics.
- Prefer counters that cannot wrap in any realistic lifetime:
snmpEngineTimefor uptime where the agent implements it properly (it counts whole seconds, so it lasts 68 years), and the 64-bit HC counters for traffic. - Never infer an event from one derived signal when you have a direct one. Uptime is an inference about reachability; reachability data is the thing itself.
- Anything that feeds money or contracts deserves a shadow run before release.