Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Providers: Check your BMCs!
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Providers: Check your BMCs!

Hello LowEndTalk Providers,

Please make sure to check your SuperMicro IPMI and related BMCs, and this most recently disclosed exploit is just one of many issues core to the management interface we all take for granted in controlling our physical assets.

As was explained on cari.net some time ago, any IPMI found on a series X9 or older motherboard from SuperMicro may be vulnerable to an exploit which can reveal the [b]plaintext passwords[/b] of [b]all users[/b].

http://blog.cari.net/carisirt-yet-another-bmc-vulnerability-and-some-added-extras/

Numerous other exploits are also found on these X9 and older BMC units, the worst of which can be used to install back-doors that [b]survive a complete host Operating System reinstall and firmware update.[/b]

IPMI and BMC devices should never have public addresses to begin with, but if you must; make sure that you are not allowing connections to these devices on ports other than those specifically designed for their functions, 80/443 ; 623 ; and 5900 for the web interface, media interface, and iKVM respectively.

Please note that in situations in which multiple customers are on the same private network, the issue is just as dear as if the IPMI were placed on a public network. Proper ACLs should be in place to prevent customers from connecting to each-other's IPMI devices, such as the use of VLANs.

If you are not yet implementing an http://en.wikipedia.org/wiki/Out-of-band_management network, then you should be. Many providers here on LowEndTalk have networks which currently contain vulnerable devices, and should seek options of dealing with this issue immediately.


Testing Vulnerable Devices

You can check if a device is vulnerable by issuing the following command:

nc -z -w5 $IP 49152

A possibly vulnerable device will return the following:

Connection to 10.0.0.254 49152 port [tcp/*] succeeded!

While a device that is not vulnerable will not return anything after 5 seconds.

Once a possibly vulnerable device has been identified, one should note that this port shares common usage with other protocols, and could be a different device or system/software listening on the given address. The following command can be used to confirm the vulnerability:

wget "$IP:49152/PSBlock" -O /dev/null -T 3 -t 1 -nv

A vulnerable device will return a line such as:

2014-06-22 19:29:10 URL:10.0.0.254:49152/PSBlock [10240/10240] -> "/dev/null" [1]

Patching Vulnerability

First, if the IPMI device was accessible to the public network, you should remove this access. If you have physical access to the infrastructure in question, steps should be taken to ensure that the ports used for IPMI connectivity are VLAN'd and ACL'd off of the public network, and should be access only through a privileged VPN or other Out of band management network.

Secondly, you should check your firewall and network logs to see if access to any devices you control may have been leaked, this is a very important step; as otherwise you would not know the extent of the breach.

Third, you should flash the firmware of the IPMI device with the latest blob available from SuperMicro, or from your SuperMicro authorized reseller/dealer, and reset it to factory defaults. You should then manually re-configure the device.

If you are unsure of what version of blob to use, ask your SuperMicro authorized reseller/dealer; they should be well aware of this exploit and be able to guide you in the installation of this required update.

If your device does not yet have a newer blob available, you should consider keeping the IPMI device offline or unplugged until a suitable blob to patch this exploit is available for your hardware.

A temporary shield against this particular issue can be carried out on Cisco routing equipment with commands similar to the following:

access-list xxx extended deny tcp yy.yy.yy.yy zz.zz.zz.zz any eq 49152
access-list xxx extended permit ip any any
access-group xxx in interface inside

Where xxx is a free access ID, and yy.yy.yy.yy is the IPMI device IP address, or IPMI device range; and zz.zz.zz.zz is the applicable subnet mask for the range you wish to apply this rule. This specifically blocks only the port 49152, and should not be considered a full solution in any case.


As always, you should follow security best practices and update all software found on your machines, and run scans such as maldet to make sure that no malicious software has been implanted whilst the exploit was around.

You can perform a maldet scan by following these steps:

cd /opt
wget http://www.rfxn.com/downloads/maldetect-current.tar.gz
tar -xvf maldetect-current.tar.gz
cd maldetect-*
./install.sh

At this point, you should drop into a terminal multiplexer, script, or screen session; as the scan can take several hours, and a disconnect from SSH would otherwise interrupt the scan. We suggest the use of screen as it is very easy to use. Simply type screen to enter a screen.

Once you are in your screen, simply type the following to scan your root partition:

maldet -a /

Comments

  • RaymiiRaymii Member

    If a provider had their BMC's on the public internet or reachable without a special admin vpn they deserve to be screwed over. Those things are a big exploit on themselves already, they provide full remote administration of a box..

    I always, always create a special management vpn for routers, switches and bmc's, which had no internet access and requires two-factor login...

    Thanked by 1linuxthefish
  • @Raymii said:
    If a provider had their BMC's on the public internet or reachable without a special admin vpn they deserve to be screwed over. Those things are a big exploit on themselves already, they provide full remote administration of a box..

    I always, always create a special management vpn for routers, switches and bmc's, which had no internet access and requires two-factor login...

    I strongly agree that no provider should ever offer fully public access to their BMCs. That being said, there are times where a VPN with two-factor may not be possible [ such as when handing BMC off to a client. ] That being said, there are still a number of security measures that can be taken [ such as internal proxy redirection ] do handle this, without exposing the BMC to the public network directly.

  • cassacassa Member

    A friend of mine found this already on the OVH servers

  • ZEROFZEROF Member

    I asked online.net about this few days before. They don't have this issue.

  • For people running Juniper gear, this little snippet will get the same results -- http://paste.ee/p/iR2At#0QRz1TelB396ofwVFyMCAfhezt7cIHbl

    You should still patch up even if you drop access, though.

  • @Wintereise said:
    For people running Juniper gear, this little snippet will get the same results -- http://paste.ee/p/iR2At#0QRz1TelB396ofwVFyMCAfhezt7cIHbl

    You should still patch up even if you drop access, though.

    Agreed, it's meant only as a temporary shield in case a patch for your specific BMC module is not yet available, or considering that it's Sunday; if you were not yet able to get a hold of your SuperMicro authorized reseller/dealer. Thanks for the Juniper-based contribution!

  • Bump, as apparently many providers still haven't done this.

  • @GoodHosting, It appears that this may have impacted SemoWeb also as they just sent out an email that reads amazingly similar to the URPad one (i.e., "8 systems were wiped which caused...").



    Thankfully, I have no existing services with them.

  • CoreyCorey Member

    GoodHosting said: Bump, as apparently many providers still haven't done this.

    Thanks - I checked mine and we're clear.

    Raymii said: If a provider had their BMC's on the public internet or reachable without a special admin vpn they deserve to be screwed over. Those things are a big exploit on themselves already, they provide full remote administration of a box..

    I always, always create a special management vpn for routers, switches and bmc's, which had no internet access and requires two-factor login...

    Well here is the issue with that, if you rent a dedicated server from another provider and don't actually own the hardware then you are at the mercy of how that provider provisions IPMI

  • dnwkdnwk Member

    Does this bug apply to Dell CS24-SC BMC?

  • qpsqps Member, Host Rep
    edited June 2014

    dnwk said: Does this bug apply to Dell CS24-SC BMC?

    No, it does not appear to. Tested against firmware 1.95.

    Thanked by 1geekalot
  • dnwk said: Does this bug apply to Dell CS24-SC BMC?

    No.

  • @Corey said:
    Well here is the issue with that, if you rent a dedicated server from another provider and don't actually own the hardware then you are at the mercy of how that provider provisions IPMI


    And this is what makes me nervous about going with VPS providers that rent dedicated servers or with renting a dedicated server vs colocation. Of course people do also setup IPMI with colo; but I am really starting to like the approach used by some (like Dacentec) for the (free) on-demand IPKVM's. As long as they set it up quickly.


    I know remote access to BIOS & ISO etc is a necessary evil; but with great power, comes great responsibility.

  • To all those asking "Is my XYZ vulnerable" ; what about testing it? I've given you all the fuel required to test the exploit on your own hardware to check the vulnerability, as well as various means of patching it.

  • dnwkdnwk Member

    @GoodHosting said:
    To all those asking "Is my XYZ vulnerable" ; what about testing it? I've given you all the fuel required to test the exploit on your own hardware to check the vulnerability, as well as various means of patching it.

    Because my server is still on its way to DC. So I have no where to test.

  • CoreyCorey Member

    geekalot said: And this is what makes me nervous about going with VPS providers that rent dedicated servers or with renting a dedicated server vs colocation. Of course people do also setup IPMI with colo; but I am really starting to like the approach used by some (like Dacentec) for the (free) on-demand IPKVM's. As long as they set it up quickly.

    I know remote access to BIOS & ISO etc is a necessary evil; but with great power, comes great responsibility.

    We do both currently. Our Dallas location is owned hardware and colocated.

  • From our contacts, there are still numerous hsots that post here that still have open and vulnerable IPMI. Why are people being so clueless? It's not that hard to update IPMI, it can even be done en masse using their API if you were competent enough to write a script for it.

    And you guys wouldn't believe which huge companies are still vulnerable, I'm not saying a thing until they patch it; but I can only phone them so many times a day to bitch about them not doing it...

  • Right, in the morning; we're going to finish our last sweep of networks we're authorized to scan; past that, it's up to you [the provider] to check your BMC's. We'll be disclosing another related exploit to localhost in the morning, or slightly later if hosts are having some trouble patching.

    Get your damned BMC's updated. It's really not that hard. I wouldn't want to make targets out of the hosts we know have this issue and refuse to fix it, but we wouldn't want customers going to these providers and getting their data stolen or lost either...

  • RaymiiRaymii Member

    If they don't update it is their own fault. You've done your best to inform them, if they don't act then they are on their own..

Sign In or Register to comment.