The wrong IDs, a permanent delete, and two weeks to put it back
A script meant to remove a retired app from customer sites was given the IDs of the sites themselves, and in 23 minutes it permanently deleted 883 sites belonging to 775 customers. The backups were fine, but Atlassian's restore tooling was built for one site at a time, so it had to invent a bulk restore during the incident, and the last sites came back 13 days later.
Times are approximate, in UTC, from the sources at the end. The diagram is simplified.
At 07:38 UTC on Tuesday 5 April 2022, a maintenance script began deleting Atlassian cloud sites, one after another. Twenty-three minutes later it had deleted 883 of them, belonging to 775 customers. Their Jira, Confluence, Atlassian Access, Opsgenie and Statuspage became unavailable. Nothing alerted, because to Atlassian’s systems this was a normal, successful deletion. The first customers got their sites back three days later. The last waited 13 days.
How an Atlassian site is put together
To a customer, a site is an address like name.atlassian.net with some products behind it. Inside Atlassian, the site is a container: the thing that ties a customer’s licensed products together. And it doesn’t live in one place.
Atlassian’s cloud is multi-tenant. Each service serves many customers, and each shard of a data store holds data for many tenants, kept apart logically. Creating a site touches a long chain of systems: commerce and licensing records, the site’s core metadata and configuration, identity data (users, groups, permissions), a database for each product, media such as attachments, and any third-party apps the customer has installed. Those systems refer to the site by identifiers such as its cloud ID, which are treated as immutable, so other records embed them freely.
Deleting things is a normal part of that lifecycle. Atlassian’s provisioning system handles events for a whole tenant (creation, destruction) and for products within it (activation, deactivation, suspension). There are two ways to delete data, and the difference matters here. A soft delete marks a record as deleted: it disappears from the product but is still there, and a later job purges it once a retention period has passed. A hard delete removes the data now. The script used for this job could do either: according to Atlassian’s CTO, it had a mark-for-deletion mode for routine work, where recovery is wanted, and a permanent-delete mode for when data must be removed for compliance.
Atlassian’s backups were in good shape. Its databases take full and incremental backups that can be restored to any point in time within 30 days, the backups are immutable, and the databases keep synchronous standby replicas in multiple Availability Zones. Atlassian regularly rolled back individual customers, or a few at a time, who had deleted their own data, and could restore every customer at once into a new environment. What it had not automated was restoring a large group of customers into the live environment, with everyone else still running on the same stores.
What happened
In 2021 Atlassian had folded a standalone app, Insight – Asset Management, into Jira Service Management as a built-in feature. The old app had to be removed from customer sites that still had it installed, and engineers used an existing script and process to do it.
Two things went wrong. First, a communication gap: the team that requested the deletion gave the team running it the IDs of the entire cloud sites where the app was installed, not the IDs of the app. Second, the deletion API accepted both kinds of ID and assumed its input was right. Pass it a site ID and it deleted the site; pass it an app ID and it deleted the app. Nothing asked which was meant. In an update a week into the outage, Atlassian’s CTO said the script had also run in the wrong mode: permanent deletion rather than mark for deletion.
The safeguards that existed didn’t see it. The script went through the standard peer review, which checked which endpoint was being called and how, not whether the IDs referred to apps or sites. It was tested in staging, but the production IDs didn’t exist in staging, so the test could not have caught them. A first production run against 30 sites went perfectly: its IDs had been gathered before the mix-up and were real app IDs.
The second run used the new list. It started at 07:38 and worked through 883 sites in order, deleting the first one moments after it started, and finished at 08:01. The first support ticket from a customer arrived at 07:46. At 08:17 Atlassian started its major incident process, at 08:24 the incident was escalated to critical, and at 08:53 the team confirmed that the ticket and the script were connected, and that this was neither an attack nor malicious. Within an hour of the script finishing, they knew exactly what had happened and exactly which sites were affected.
Knowing didn’t help as much as it should have. The script had also deleted key customer identifiers and contact details, such as site URLs and the site administrators’ contacts. Atlassian’s support form required an Atlassian ID and a valid site URL, so most affected customers couldn’t file a ticket the normal way, and Atlassian couldn’t easily reach them either: many of the billing and technical contacts it still had were out of date. Early estimates were about 700 sites (883 less Atlassian’s own) and about 400 customers; by Atlassian’s official definition the final count was 775. At 12:38, once the complexity of the restore was clear, the incident was given the highest severity.
Why it took two weeks
The first approach, Restoration 1, treated each deleted site as a new one to rebuild. It began at 09:37 on day one and ran to about 70 steps, mostly in sequence: create a new site with its licences and a new cloud ID, move it to the right region, restore and re-map the core metadata, restore and re-map identity data, restore the product databases, re-map attachments, set feature flags, re-map data across every other service and the third-party apps, then validate, first by Atlassian and then by the customer.
The re-mapping was the slow part. A new site means new identifiers, and because identifiers were supposed never to change, they were embedded in large amounts of data, including data held by third-party app vendors, who had to be coordinated with site by site. Restoration 1 worked. The first restored and validated site went back to its customer at 03:58 on 8 April. Once optimised, a batch took about 48 hours, and at that pace restoring everyone would have taken three weeks.
On 9 April the team proposed Restoration 2: instead of rebuilding sites, un-delete them. Recreate the deleted records in every system, starting with the site’s record in the Catalogue Service, and reuse all of the old identifiers. That removed more than half of the steps, including all the mapping from old identifiers to new and the round trips to app vendors. It had more than 30 steps, and far more of them could run in parallel. The first Restoration 2 run started at 00:56 on 10 April.
Switching approaches in the middle of an incident had its own cost. Scripts built for Restoration 1 had to be changed, batches ran in both approaches at once, and Restoration 2 had to be tested and validated before it could be scaled, repeating validation work already done for the first approach.
Each site was hard for the same underlying reason: the data stores were shared. The deleted data was a slice of stores that other customers were still using, so each site’s pieces had to be extracted from backups individually. For most customers the team picked a restore point five minutes before the deletion as a safe synchronisation point for the main data stores, and restored the other stores to the same point or replayed their recorded events, so that everything was consistent. For 57 customers restored early, inconsistent policies and manually retrieved snapshots meant some Confluence and Insight databases went back to an earlier point; a later audit found it, and the missing data was recovered.
The other critical path was people. Every restored site had to be checked, and to keep up, Atlassian organised more than 450 support engineers into shifts covering 24 hours a day to run validation. The team froze all engineering changes from 03:30 on 8 April to avoid inconsistencies in customer data and unrelated incidents. Engineers built tooling to front-load the slowest steps, database restores and identity synchronisation, and to validate restored sites automatically. Every site was technically restored, with automated validation complete, at 10:05 on 16 April, and full restoration was confirmed to customers at 01:00 on 18 April.
Communication lagged behind the engineering. The first Statuspage update came at 09:03 on day one, and the root cause was confirmed there at 11:13, but the first broad public acknowledgement, a tweet, came at 00:56 on 7 April, and an apology from the co-CEO was emailed to affected customers at 01:50 on 8 April. Atlassian’s own review says it should have communicated widely much earlier, and been open about what it didn’t yet know, instead of waiting for a complete picture.
How they fixed it
In the end Atlassian met its recovery point objective, the limit on how much recent data may be lost, which was one hour: no customer lost more than five minutes. It missed its recovery time objective, the limit on how long a restore may take.
Bulk site deletions were blocked immediately. The review then set out four areas of change. Soft deletes become universal: deleting a whole site is to be prohibited, customer data and metadata can only be deleted after a soft delete, deletions only follow deactivation, and any new deletion runs first on Atlassian’s own sites, then on customers in stages, with a tested rollback plan. The disaster recovery programme is to automate restoring many sites with all their products, and to rehearse it regularly at that scale. The incident process gets a playbook, tooling and simulations for incidents with hundreds of people working in parallel. And communications get a playbook too, with key contacts backed up outside the product and a way for customers without a working site URL or Atlassian ID to reach support.
A backup is only as useful as the smallest thing you can restore from it quickly. Atlassian could restore a single data store, a single site, or everyone into a new environment. It had no fast way to restore hundreds of sites into a live system, and that was the case its customers needed.
What I'd take from it
- Make the type of an ID impossible to confuse. The API took a site ID or an app ID and deleted whatever it was given. A separate endpoint per type, a typed prefix, or a lookup that confirms what an ID refers to turns a wrong list into an error instead of a deletion.
- Soft delete first, always. Marking data as deleted and purging it after a retention period costs some storage and buys an undo. Keep permanent deletion for compliance jobs, behind separate permissions, not as a mode of the routine clean-up script.
- Review the data, not only the code. The peer review checked which endpoint was called and how, and staging couldn't catch the problem because the IDs didn't exist there. For a destructive job, print what it will do with the real production list and have someone who knows the data read it.
- Design restores at the unit you will lose. Point-in-time backups of shared databases worked. Putting a few hundred tenants back among thousands of live ones had no tooling. Decide the unit you will restore, a tenant or a site, and rehearse restoring hundreds of them at once.
- Keep the contact list outside the blast radius. The script also deleted the site admins' contact details, and the support form needed a valid site URL. Store who to call, and a way for them to reach you, somewhere the incident can't touch.