All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
SoftBank-owned IDCF Cloud Hit by Ransomware: East-1 Wiped
I just lived through what might be every sysadmin’s worst nightmare, courtesy of Japan's enterprise cloud provider IDC Frontier (IDCF), a major subsidiary of SoftBank.
I’m writing this because we often preach the 3-2-1 backup rule here on LET, but watching an entire availability zone and its hypervisor-level snapshots get vaporized in real-time is a whole different level of educational pain.
The Incident: A Four-Day Gaslight
From October 2 to October 6, IDCF’s management backend started throwing errors: internal repo mirrors went down, and provisioning APIs for DNS, DBs, and compute failed.
IDCF sent out four consecutive updates reassuring customers that this was purely a "hardware failure in the operational physical infrastructure," explicitly stating there was "no unauthorized access or cyberattack". They promised a full fix by October 7 at 09:00 JST.
Like many customers, I assumed a massive telecom-backed enterprise provider knew what they were doing. I didn't rush to dump my volumes off-site.
The Reality: Hypervisor Wipeout on East-1
On October 7 at 03:40 JST, the entire East Japan Region 1 (tesla, henry, pascal, and joule zones) went pitch black. Virtual machines across all four clusters were forced offline.
My primary node was hosted on henry.
By that afternoon, the official narrative completely collapsed:
- The Cause: Confirmed external ransomware attack targeting the cloud management infrastructure.
- The Blast Radius: East-1 was physically severed from the network.
- The Permanent Loss: IDCF officially admitted that virtual machines and storage volumes in East-1 are virtually unrecoverable.
- The Snapshots: Even if you paid for IDCF's daily platform snapshots, they cannot be extracted or restored. If you didn't keep an independent off-site backup outside IDCF, your data is gone forever.
- The Advice: IDCF explicitly advised affected customers to rebuild on third-party infrastructure and warned not to rebuild inside IDCF’s other regions.
The Domino Effect on the Rest of the Platform
To prevent lateral infection, IDCF hit the panic button and pulled the plug on the web console and API nationwide.
This left surviving instances in East-2/3 and West Japan running completely blind:
- Scheduled snapshots: Completely dead.
- Integrated Veeam backups: Inaccessible.
- Database failover (Active/Standby): Broken.
- Autoscaling & DNS management: Paralyzed.
IDCF had to issue emergency CLI guides instructing surviving customers to SSH in and manually rsync their databases out before anything else went wrong.
As of today, the customer ticket portal is completely dead, emergency inquiries are handled via a lone email box ([email protected]), and the Tokyo Metropolitan Police Department (Keishicho) is on-site investigating.
Hard Lessons from a Dead Node
- Provider snapshots are not isolated backups: If the hypervisor control plane or centralized SAN metadata pool gets encrypted, cloud-native snapshots die right alongside the live block storage.
- Never trust early outage PR: When a host claims "purely a physical hardware defect" while their central API and template pools are completely locked down, treat it as an active compromise and pull your data immediately.
- The enterprise illusion: Having a conglomerate logo on the invoice doesn't change the standard cloud contract. Limitation of liability clauses mean no cloud provider is writing checks for operational downtime or lost business.
Curious to hear from the community: how many of you actively keep automated, out-of-band immutable cold backups (e.g., S3 Glacier with Object Lock / off-site ZFS receive), and how many are still relying on your provider's built-in snapshot manager?
Comments
@FAT32
I can feel their pain.
When shit happens, it can happen to anyone, no matter how big or secure the provider is.
As a client, always keep your own backups in multiple locations if your data is important.
I feel the coming months or years will be rough, and we may see more incidents like this. Blaming the provider won't bring back lost data. Of course providers have responsibilities, but these days even entire datacenters are getting attacked, like in Ukraine.
Even if a host is secure, shit can still happen, like a datacenter catching fire. So never depend on one provider for your backups.
This is probably the largest ransomware hit of all time
Actually I am not aware of any host this size to be hacked like this with core data access even non ransomware attacks
I always have automated offsite backups, either with a locked down rrsync or more usually S3+object lock, and hosted with a different provider in a different country/region.
Snapshots or provider backups can be convient, but cannot be relied upon. Bad actors, natural disasters, power issues, network issues, disputes with provider, payment issues...
that's really sad.
providers should keep backup/snapshot stored on other provider's dc in some other country. and same for the users too. I have seen many people having aws instance and saving backup in same region/dc. avoid that.
My suggestions:
Get onsite server:
Archive WAL and PITR (adjust according to down time tolerance)
Make offline backups:
Archival quality disks, tapes, drives stored in secure location. I recommend safe deposit boxes at high rise building (since more floors == more sacrificial plates for tactical missile strike)
Make ransomware drills
Rehearse total data loss on production
so now you tell me 1 cloud provider with multiple zone is not enough, they selling point saying HA but when one zone down it affecting the whole region.
So keep paying more, and remember to backup offsite.
Softbank's companies has a subscription to failures.
Yes, correct. Sad that still many do not understand that.
Wrong. Never trust ANY PR!
Yes, correct. Sad that still many do not understand that.
Btw, Kassa, I'm sorry that you had to learn that lesson the really hard way.
Many successes make up for the major losses.
That's also false safety. They have their credentials and infrastructure hacked. Just because it's in another DC doesn't mean it's not accessible from the hacked network. In fact, it's more likely to be whitelisted or securities bypassed.
Quick update for anyone following the IDCF situation — they just released Reports #20 and #24 with a surprisingly positive turnaround after days of radio silence:
Looks like the ~40% who paid for automated cloud snapshots dodged a total wipeout[cite: 2], though the other ~300 without snapshots might still be out of luck.
@KASSA do you have snapshot or not?