New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Can we say that this incident was malicious negligence? After all, ipmi via public access is beyond dumb. Or just the standard lack of competence?
We've provided all updates, customer issue has been resolved and any known vulnerability impact has been patched. Not all services have public facing IPMI ports, many do not.
Hi,
i know about Dell iDRAC getting infected too, at least the older ones. ( Version 7 ).
HP iLO 4 and 5 seems to be more resilent -- so far didnt hear of that. But they have a quiet restrictive timeout enlargement by default for failed logins.
We'll be implementing a VPN/DCIM based IPMI access network for any custom based dedicated servers as standard practice moving forward. This is our current setup on instant deploy dedicated servers.
My two cents here. It's not the responsibility of the provider to secure publicly accessible services on your co-located and/or rented gear.
If this was their own management panel, then yeah they would have some responsibility for vulnerabilities.
IP-restrict your IPMIs, people. I have a dedi at fiberstate, with a public IPMI that's only open to specific IPs that I added in Settings. I won't have any secure resource open to the public. If I had a server with an IPMI that couldn't firewall itself, and the provider's network team couldn't IP-restrict traffic, then I would cancel the service.
I'll hop off my soap box now.
mine is instant deploy dedicated server, so i hope it is not impacted
Nothing is impacted at the moment. We've also explained this in detail in many of our posts. If you do have any concerns or would like us to double check, please feel free to open a support ticket.
But is that relevant or is this:
So @fiberstate I appreciate the problem you’re facing and I very much empathize with staring down an old problem that is bigger than you. Been there, won’t be the last time. But as someone who is always window shopping and has peaked in your window many times I have to say the responses to this have been subpar. We’ve got two users here on your network with this happening to them, who knows if there’s more. At least one I see saying it’s a rented dedicated server. Even if it wasn’t, I don’t agree that it’s not your concern if IPMI is exposed on your network. Here are a couple of key points in my mind right now:
It’s your network and a compromised server is a risk to it. For the same reason you would deny certain types of services the right to run on your network (as any good provider would), a public facing IPMI should be among them.
You’re very quickly brushing off the fact that you are deploying servers with public facing IPMI. This is where you should be accepting responsibility and telling customers you won’t rest until the wrong is right. Instead you’re calling attention to a customer mistake and glossing over the big one: a customer should be able to make the password 123 because it shouldn’t be on the public internet.
You do with that what you will. I’m about to get up and wipe, I said what was on my mind.
We are in agreement and have mentioned above that our IPMI deployment will be changed moving forward. How it will be implemented is to be determined. We have also stated that there are current built-in firewall options available to users, that can be set to limit IP access. Users can also request we disable IPMI while we place the IPMI connection behind a VPN access portal. We are sending these notices out now. We do feel confident that the Asrock Rack B650D4U boards we have in use have all been secured.
When we did find a manufacturer issue with IPMI in the particular model (B650D4U) motherboard, we've physically changed every single one to prevent any further issues. So leading credence to your comment "you won’t rest until the wrong is right", we very much believe this is the only way to be an effective service provider and we are doing such. We've also comped the impacted user in the post a free month of service for their troubles.
We said exactly that in one of our last replies.
"We'll be implementing a VPN/DCIM based IPMI access network for any custom based dedicated servers as standard practice moving forward. This is our current setup on instant deploy dedicated servers."
So yes, changes will be made and are forthcoming to our current IPMI deployment practices.
And this is why I refuse all kinds of remote management access (SSH AND ILO4) being public at my home lab. VPN or get stuffed.
SSH on VPSes? Setup IPAddress accounting + SSH keys.
Locking down isn't hard.
@jbiloh can we ban @fiberstate and @Shakib ? Or at least revoke the provider tag.
They keep exposing IPMI's to the internet knowingly.
This shit is nuts.
Wait until they get 22k in crypto. :P
Scamming 22k no ban ✅ exposing public ipmi ban ❌
@Neoon, We've taken every corrective step possible, changed many servers to ensure there is no threat to their IPMI and are going a step further and pulling any IPMI to be placed behind a private VPN. The one user impacted by this asrock B650D4U IPMI user issue has been fully secured, reloaded and compensated.
Standard deployment practice for any IPMI on our network moving forward will be provided to the end user with a VPN. In addition, this is already in practice with most servers that have been deployed on our network.
While it is not responsibility per-se, but it is very good indication of secops culture of the provider. Today it is IPMI via public IP, tomorrow it is SSH with default password. Where is the end? Publicly accessible node backups?
There is providers who just do not care. All they care is money. And a quick one.
We are most certainly NOT that provider. This is a real jump to go there, we do not and have never deployed a single server with a easy-to-guess or default SSH password.
DRAC and iDRAC have common default passwords, iLO 4 and 5 do not, the unique default password is on the case.
May or may not be a factor..
We always ensure a random, 15 or more character password is set with every possible character set combination.
You done your job, you fix your mess. This is not about you. We discuss in general how it is in industry. Consult with your marketing manager before posting publicly if you feel that emotions takes best of you. So far, you handle communication and issue with good taste. Do not go dark side.
I am not a moderator, I suggested it.
Even thinking about liability, what insurance is gonna cover that?
We've discussed what our implementation changes are and will be, this user issue has been patched on all known instances on our network for the motherboard model ASrock Rack B650D4U.
Standard deployment practice for any IPMI on our network moving forward will be provided to the end user with a VPN. In addition, this is already in practice with most servers that have been deployed on our network using DCIM. We are in the process now of changing any remaining servers over to this and or a VPN system for IPMI access.
Bottom line is we do own this issue on IPMI deployment and are implementing a change in IPMI security practices we use moving forward. We value this community and the important feedback it provides. We will do better in this regard.
What about iDRAC9? - it's also on public ip with Fiberstate afaik.
(Even though this vuln is not about iDRAC, deploying the VPN solution for iDRAC would be nice as well.)
We are implementing a full network-wide audit and moving any IPMI/iDrac private. Please feel free to open a ticket if you need assistance.
Well, with IDRAC, as well as any other remote hands, you can create ACLs that will drop 0.0.0.0 and allow your IP.
Could be @fiberstate fault for not providing a VPN connection? Kinda.
Could it be your fault for not taking all security measures you could take and set up ACLs? Absolutely!
Correct. Unless the server is a instant deploy in which case its already on a private network and do not set a IPMI firewall block, it will disable DCIM functionality.
We are implementing a full network-wide audit today and moving any IPMI/iDrac private with VPN/DCIM access.
Please don't do those type of DCIM-WHMCS integration where you get IPMI access via a Virtual Machine, rather than running the java applet on your own PC. Had numerous bad experiences with that including, but not limited to:
... and many more.
We use easydcim currently, looking at testing tenantos for our Los Angeles deployment.
It's the same shit honestly... tenantos works the same way.
Could you, at least, let existing (custom) deployments out of this system?
Custom deployments are completely stand alone and manually provisioned.
From this thread.. Open a ticket and they will change it for you..