All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
SoftBank-owned IDCF Cloud Hit by Ransomware: East-1 Wiped
I just lived through what might be every sysadmin’s worst nightmare, courtesy of Japan's enterprise cloud provider IDC Frontier (IDCF), a major subsidiary of SoftBank.
I’m writing this because we often preach the 3-2-1 backup rule here on LET, but watching an entire availability zone and its hypervisor-level snapshots get vaporized in real-time is a whole different level of educational pain.
The Incident: A Four-Day Gaslight
From October 2 to October 6, IDCF’s management backend started throwing errors: internal repo mirrors went down, and provisioning APIs for DNS, DBs, and compute failed.
IDCF sent out four consecutive updates reassuring customers that this was purely a "hardware failure in the operational physical infrastructure," explicitly stating there was "no unauthorized access or cyberattack". They promised a full fix by October 7 at 09:00 JST.
Like many customers, I assumed a massive telecom-backed enterprise provider knew what they were doing. I didn't rush to dump my volumes off-site.
The Reality: Hypervisor Wipeout on East-1
On October 7 at 03:40 JST, the entire East Japan Region 1 (tesla, henry, pascal, and joule zones) went pitch black. Virtual machines across all four clusters were forced offline.
My primary node was hosted on henry.
By that afternoon, the official narrative completely collapsed:
- The Cause: Confirmed external ransomware attack targeting the cloud management infrastructure.
- The Blast Radius: East-1 was physically severed from the network.
- The Permanent Loss: IDCF officially admitted that virtual machines and storage volumes in East-1 are virtually unrecoverable.
- The Snapshots: Even if you paid for IDCF's daily platform snapshots, they cannot be extracted or restored. If you didn't keep an independent off-site backup outside IDCF, your data is gone forever.
- The Advice: IDCF explicitly advised affected customers to rebuild on third-party infrastructure and warned not to rebuild inside IDCF’s other regions.
The Domino Effect on the Rest of the Platform
To prevent lateral infection, IDCF hit the panic button and pulled the plug on the web console and API nationwide.
This left surviving instances in East-2/3 and West Japan running completely blind:
- Scheduled snapshots: Completely dead.
- Integrated Veeam backups: Inaccessible.
- Database failover (Active/Standby): Broken.
- Autoscaling & DNS management: Paralyzed.
IDCF had to issue emergency CLI guides instructing surviving customers to SSH in and manually rsync their databases out before anything else went wrong.
As of today, the customer ticket portal is completely dead, emergency inquiries are handled via a lone email box ([email protected]), and the Tokyo Metropolitan Police Department (Keishicho) is on-site investigating.
Hard Lessons from a Dead Node
- Provider snapshots are not isolated backups: If the hypervisor control plane or centralized SAN metadata pool gets encrypted, cloud-native snapshots die right alongside the live block storage.
- Never trust early outage PR: When a host claims "purely a physical hardware defect" while their central API and template pools are completely locked down, treat it as an active compromise and pull your data immediately.
- The enterprise illusion: Having a conglomerate logo on the invoice doesn't change the standard cloud contract. Limitation of liability clauses mean no cloud provider is writing checks for operational downtime or lost business.
Curious to hear from the community: how many of you actively keep automated, out-of-band immutable cold backups (e.g., S3 Glacier with Object Lock / off-site ZFS receive), and how many are still relying on your provider's built-in snapshot manager?
Comments
@FAT32
I can feel their pain.
When shit happens, it can happen to anyone, no matter how big or secure the provider is.
As a client, always keep your own backups in multiple locations if your data is important.
I feel the coming months or years will be rough, and we may see more incidents like this. Blaming the provider won't bring back lost data. Of course providers have responsibilities, but these days even entire datacenters are getting attacked, like in Ukraine.
Even if a host is secure, shit can still happen, like a datacenter catching fire. So never depend on one provider for your backups.
This is probably the largest ransomware hit of all time
Actually I am not aware of any host this size to be hacked like this with core data access even non ransomware attacks