fin1te

Writing/·5 min read

How a cloud SLA is actually calculated

99.95% availability leaves about 22 minutes of downtime a month. Deciding which minutes count is the hard part: HA pairs, reboots nobody saw, outages that span a weekend, maintenance windows and assets that did not exist yet.

SLASparkScalaMonitoring

A cloud SLA sounds like a single number. The provider promises 99.9%, 99.95% or 99.99% availability, and if it falls short, the customer gets service credits. Over a 30-day month those promises leave very little room:

Commitment Downtime allowed per month
99.9% 43.2 minutes
99.95% 21.6 minutes
99.99% 4.3 minutes

The formula is the easy part. The hard part is deciding, for thousands of servers, VMs, storage arrays and network devices, exactly which seconds were downtime, which of those count, and proving it afterwards. Get it wrong one way and you pay penalties for outages that never hurt anyone. Get it wrong the other way and you under-report, which is worse.

In July and August 2025 I built the engine that does this for a cloud platform: a daily Spark job, written in Scala, that turns raw monitoring data into an auditable downtime record per asset, per day. This is how it works.

The inputs

Every night the job processes the previous day from four sources:

  • Monitoring time series. Zabbix polls an uptime counter on every host and device.
  • Problem events. Node-down events raised by ICMP ping triggers.
  • The CMDB. When each asset was provisioned and retired, what category it belongs to, and whether it is in contractual scope.
  • Change tickets. Approved maintenance windows from the ITSM system, with planned start and end times.

1. Finding restarts in uptime counters

An uptime counter only goes up while a machine is running. If it goes down between two samples, the machine restarted. A window function finds those drops:

SELECT itemid,
       FROM_UNIXTIME(prev_clock)        AS down_start,
       FROM_UNIXTIME(clock - value)     AS down_end,
       (clock - value) - prev_clock     AS down_seconds
FROM (
  SELECT itemid, clock, value,
         LAG(clock) OVER (PARTITION BY itemid ORDER BY clock) AS prev_clock,
         LAG(value) OVER (PARTITION BY itemid ORDER BY clock) AS prev_value
  FROM history_uint
  WHERE itemid IN (/* uptime items */) AND DATE(FROM_UNIXTIME(clock)) = :day
) h
WHERE prev_value > value
  AND (clock - value) > prev_clock;

The neat part is the end time. The first sample after the restart says how long the machine has been up, so clock - value is the moment it came back, even if the next poll happened minutes later. The outage runs from the last good sample to that moment.

A counter can also drop without a restart: 32-bit uptime counters wrap after about 497 days. Any drop from near that ceiling is only kept if ICMP ping also failed during the same window. That guard turned out to matter more than expected, and it got its own write-up.

2. High availability: full or partial

A lot of the estate runs in HA pairs: active-passive nodes, redundant controllers, clustered databases. If one node of a pair goes down and the other takes over, the customer sees nothing. Charging that as an outage would penalise the provider for redundancy doing its job.

So every clustered asset gets a label:

  • Standalone asset down: full outage.
  • One HA node down, partner up: partial outage. Degraded, but no SLA breach.
  • Both HA nodes down at the same time: full outage.

The tricky case is overlap. Node A is down from 10:00:12 to 10:07:40 and node B from 10:05:03 to 10:09:00. Only 10:05:03 to 10:07:40 is a full outage. Interval arithmetic on overlapping ranges is easy to get subtly wrong, so the engine takes a blunt but reliable approach: explode each downtime window into one-second slices, join each slice against the partner’s slices for the same second, and label it.

val slices = downtime
  .withColumn("second", explode(splitIntoSeconds(col("down_start"), col("down_end"))))

val labelled = slices.as("a")
  .join(slices.as("b"),
    col("a.name") === col("b.ha_partner") && col("a.second") === col("b.second"), "left")
  .withColumn("outage",
    when(length(col("a.ha_partner")) === 0, "Full")      // standalone
      .when(col("b.second").isNotNull, "Full")            // both down
      .otherwise("Partial"))                             // partner covered

Consecutive seconds with the same label are then compacted back into intervals with a min/max aggregation. One second of resolution per host sounds expensive, but outages are rare and short, so the exploded data stays small, and the logic is simple enough to trust.

3. The outage nobody saw

If a server dies on Friday evening and comes back on Monday morning, monitoring has nothing to say in between. There is no sample to take from a dead machine. On Monday it reports again, and the engine has to work out the rest.

The daily summary stores the first uptime reading of each day. On the recovery day, subtracting that uptime from the time of the reading gives the true recovery time, even if monitoring only picked the host up hours later. (In Oracle date arithmetic that is first_reading - uptime_seconds / 86400, because dates subtract in days.)

The span from Friday to Monday is then split into one row per calendar day:

val perDay = outages
  .withColumn("day", explode(sequence(to_date(col("down_start")), to_date(col("up_since")), expr("INTERVAL 1 DAY"))))
  .select(
    when(col("day") === to_date(col("down_start")), col("down_start"))
      .otherwise(col("day").cast("timestamp")).as("down_start"),
    when(col("day") === to_date(col("up_since")), col("up_since"))
      .otherwise(expr("cast(date_add(day, 1) as timestamp) - interval 1 second")).as("down_end"),
    col("asset"))
Day From To
Friday 14:00:00 23:59:59
Saturday 00:00:00 23:59:59
Sunday 00:00:00 23:59:59
Monday 00:00:00 10:00:00

One guard keeps this honest. If monitoring did see a real up or down transition on one of those days, the extrapolated row for that day is wrong, so an anti-join against days with real transitions removes it.

4. Assets that did not exist yet

An asset provisioned at 10:00 cannot owe anyone for an outage that started at 08:00. An asset retired at 15:00 should not be charged for alerts at 16:00, when it is being wiped. The engine joins every downtime row with the asset’s lifetime from the CMDB, drops rows entirely outside it, and clips rows that straddle either end:

.filter((col("created").isNull || col("down_end") > col("created")) &&
        (col("retired").isNull || col("down_start") < col("retired")))
.withColumn("down_start", greatest(col("down_start"), coalesce(col("created"), col("down_start"))))
.withColumn("down_end",   least(col("down_end"),     coalesce(col("retired"), col("down_end"))))

5. Planned maintenance

Downtime inside an approved change window is exempt. Downtime outside it is not, even if it is the same outage. A server that goes down at 01:30 for a change window of 02:00 to 04:00, and comes back at 05:00, produces three pieces:

01:30 ─ 02:00   unplanned   counts against the SLA
02:00 ─ 04:00   planned     exempt, tagged with the change ID
04:00 ─ 05:00   unplanned   counts: the change overran

A UDF walks the change windows for the asset in time order and cuts the outage into exactly those pieces. Where a planned and an unplanned row end up covering the same asset and time, a broadcast anti-join lets the planned one win.

6. Noise, and an audit trail

Two last filters:

  • A minute or less is dropped. Agent restarts and probe jitter produce short blips that no customer ever noticed, and they generate disputes out of all proportion to their size.
  • More than 86,400 seconds in a single row is rejected as invalid, because by this point every multi-day outage should already have been split into days. A longer row means a bug upstream, not an outage.

Nothing is ever overwritten. When a raw outage is split or clipped, the original row stays in the table marked inactive, and the derived rows are written as active. Anyone disputing a number can see exactly how it was produced.

The output

The result is one row per asset, per day, per piece of outage, each tagged full or partial, planned or unplanned, with its change ID where there is one. From there, availability is simple arithmetic: unplanned full-outage seconds against in-scope seconds, per asset and per customer.

Getting to that simple arithmetic took all six steps. Each one exists because a specific kind of wrong answer showed up in real data first.

What I would tell anyone building one

  • Derive recovery time from the uptime counter itself, not from when monitoring noticed.
  • Model HA explicitly. Redundancy is what customers pay for; the SLA should reflect it.
  • Split everything at midnight before you aggregate anything.
  • Clip to asset lifetimes, and treat planned maintenance as a cut, not a flag.
  • Never overwrite. Keep the raw row, mark it superseded, and write what you derived next to it.