Serious security issue at CloudFlare - CHANGE ALL YOUR PASSWORDS NOW.

Razza

One of the bigger host that will be affected is Digital Ocean they use Cloudflare on the main site and panel.

FlamesRunner

Man, am I lucky to have 2FA on all of my accounts and servers

I'll have my passwords changed, again...

FlamesRunner

I found an insecure domain!

Yours truly, http://lowendtalk.com

joepie91

Apparently Discord is affected too. @jarland

EDIT: Also, some people are compiling a list of sites that use CloudFlare.

ricardo

The list is going to be big as they claim to host millions of sites.

Looks like that list is just the Alexa million. Take the gTLD zone files and check those... I'm sure it'd get into the tens of thousands, ignoring subdomains.

jar

@Razza said:
One of the bigger host that will be affected is Digital Ocean they use Cloudflare on the main site and panel.

Yes this is true, panel uses CF. I'm not worried about it personally after a conversation I just had but ignore my feelings about it entirely, not my thing to share, never shy away from changing passwords. It's too easy to ever consider it the wrong action. The only one that could ever be wrong is to not.

bauhaus

Take note: my.vultr.com is using CF.
Thanks @joepie91, much obliged.

tmwc

I actually have only one site on Cloudflare. Already have Netlify for some of my sites and the others just don't have any protection/CDN. Anyways, good call, will change PW.

JahAGR

wowwwww

cociu

for the moment i will remove cloudflare from our panel .... this is a sad issue

bsdguy

Unfortunately, cloudflare chose to play it the usual corp. way and to basically say "Don't worry, chances are 1 in x millions" and "basically it was just a small detail somewhere".

Looking closer I do not believe them and I'll provide four (of possibly more) reasons:

a) "We have looked, it's OK now" - Bullshit! cloudflares capacity and capability to find and track down serious memory problems is proven to be lacking. Otherwise that issue would have come up in the first place.
Also this event shows that cloudflare did not properly test their new code before going production but they did it only now.

b)
/* generated code */
if ( ++p == pe )
goto _test_eof;

They either abused goto in a state machine without proper design and testing/verification. Plus they used lousy variable names. Plus they coded nonchalantly which according to their own explanation lead to the problem.

Concrete:

• if p+1 == pe
• instead of "pe" "buffer_end" would have been clearer.
• I'm missing a else if (p+1 > pe) panic("SCREAM and panic!").

And that is for a normal company. Some in the position of cloudflare (giants having considerable parts of the web going through them) one would have expected at the very minimum to run their generated code through a static analyzer. I know of a few which would spot that kind of problems.

c) " This indicated that we were not using Ragel correctly." - Nope. This indicates that they acted utterly careless, just trusting Ragel to create reliable and safe code and/or that they didn't even understand ragel really well.

And they STILL haven't got the message! Rather than at least now doing proper analysis they blabber about fuzzy testing and blurb proudly of that.

d) Their marketing and PR blurb (because that's how they really handle it) mentions that a very dramatic part of the problem was that their utterly unchecked ragel generated code was playing with the web servers memory! The guy responsible for that should be shot right away.

But don't you worry. It's only one in millions of cases and they did fuzzy testing now...

WSS

@bsdguy said:

See my post @17:45..

bsdguy

I feel a large wave of relief. That whitney dumbderp has informed on twitter -> "Three layers of encryption keeps you safe when SSL/TLS fails"

She was/is with eff and ftc and hacker-whatnot. She knows.

And: NO, I will not return her brain. I auctioned that for 80 cents (read: bad performance/price ratio) and I intend to keep it. After all I must find out about the 3 layers of encryption in telnet.

WSS

Well, that's the first SYN you made, @bsdguy..

jiggawatt

kcaj said: It does however raise second thoughts at running half the internet through a massive MITM reverse proxy.

As opposed to storing your email (connected to all your banks/social media accts) on a host accessible to the global Internet?

Neoon

@cociu said:
for the moment i will remove cloudflare from our panel .... this is a sad issue

Mate, the Traffic ALWAYS even with SSL is unencrypted inside the Cloudflare Network and they put Cloudflare in front of everything... thats retarded.

add_iT

virmach solusvm panel use cloudflare as i remember

UrDN

joepie91 said: This is probably a good moment to refer back to the article I wrote last year, about how CloudFlare is actively putting the web at risk.

Well, what do you expect... Cheap security through obscurity is more accessible than paying a good maintainer.

datanoise

Wow.

Many of us had the feeling that Cloudflare was a pretty nasty / dangerous thing. Sounds like it's proven right :-/

Hybrid

All my Cloudflare websites are on that list... Gonna take hours to change all the passes

dragon2611

I don't think anything I care about having hacked uses CF, Sure you could get my details out some forum I suppose.

Ole_Juul

It's basically all been said above, but to me it boils down to two philosophies. Those that think that a single point of failure is OK, and those that don't.

myhken

Do I understand correctly that this issue do not collect data from sites - aka have full access to the database etc, so if I'm member of a site on the list, but have not login to the site in the period of the security issue (mention to be between 2016-09-22 - 2017-02-18) there is no way that my password has been in risk? I understand that if I have just login to one of the sites, once, that password can be in risk. Do I understand it correct?

jh

Thanks. I'm going through important sites and trying to figure out which are behind a MITM service. Already found our savings bank is behind Incapsula. It's fair enough for forums, news sites etc. (what's the worst that could happen?) but a lot of companies ought to know better.

MrKaruppu

Sites that use CF:

Domain providers:

NameCheap

NameSilo

VPS/Server providers:

DigitalOcean

Wholesaleinternet - @AaronW

VortexNode - @VortexMagnus

VirMach - @VirMach

vpsRus - @vpsRus

InceptionHosting/ServersNV - @AnthonySmith

HostSlayer - @VENETX

BudgetNode - @Ishaq

LaunchVPS - @launchvps

AlphaVPS - @AlexBarakov

ContraWeb - @ContraWeb

VPSfast - @vpsfast

BHost - @BHost

Forums:

LowEndTalk

LowEndBox

WebHostingTalk

I will update this list as I find new reputed providers/forums.

doughmanes

Maybe the CloudFail fanboys will stop

ricardo

Dear Cloudflare Customer:

Thursday afternoon, we published a blog post describing a memory leak caused by a serious bug that impacted Cloudflare's systems. If you haven't yet, I encourage you to read that post on the bug:

https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/

While we resolved the bug within hours of it being reported to us, there was an ongoing risk that some of our customers' sensitive information could still be available through third party caches, such as the Google search cache.

Over the last week, we've worked with these caches to discover what customers may have had sensitive information exposed and ensure that the caches are purged. We waited to disclose the bug publicly until after these caches could be cleared in order to mitigate the ability of malicious individuals to exploit any exposed data.

In our review of these third party caches, we discovered data that had been exposed from approximately 150 of Cloudflare's customers across our Free, Pro, Business, and Enterprise plans. We have reached out to these customers directly to provide them with a copy of the data that was exposed, help them understand its impact, and help them mitigate that impact.

Fortunately, your domain is not one of the domains where we have discovered exposed data in any third party caches. The bug has been patched so it is no longer leaking data. However, we continue to work with these caches to review their records and help them purge any exposed data we find. If we discover any data leaked about your domains during this search, we will reach out to you directly and provide you full details of what we have found.

To date, we have yet to find any instance of the bug being exploited, but we recommend if you are concerned that you invalidate and reissue any persistent secrets, such as long lived session identifiers, tokens or keys. Due to the nature of the bug, customer SSL keys were not exposed and do not need to be rotated.

Again, if we discover new information that impacts you, we will reach out to you directly. In the meantime, if you have any questions or concerns, please don’t hesitate to reach out.

Matthew Prince
Cloudflare, Inc.
Co-founder and CEO

MrKaruppu

doughmanes said: Maybe the CloudFail fanboys will stop

I think everyone's worried that their private data might be at stake. Most of the people who commented are customers and they are concerned about their data. Doesn't look like they are happy about the CloudFlare's security issue. I would say they are concerned customers rather than cloudfail fanboys

joepie91

myhken said: Do I understand correctly that this issue do not collect data from sites - aka have full access to the database etc,

Correct. This only affects data that was sent through their proxies, which could be passively collected. It doesn't allow for eg. obtaining access to servers or the data on there, unless it was sent "over the wire".

myhken said: so if I'm member of a site on the list, but have not login to the site in the period of the security issue (mention to be between 2016-09-22 - 2017-02-18) there is no way that my password has been in risk?

Unfortunately that's not correct, however. It's basically impossible to assess scope for this kind of issue - I don't believe that the issue only affected that timespan (because it was a process issue, not just a bug), nor do I believe that this is the only such issue present (because once again, it's a process issue, not just a bug). It's also not easy to figure out who used CF when.

You should assume that everything you've sent over HTTP(S) is compromised - unless you know for a fact that a given website has never used CloudFlare, in which case that data is not at risk.

xrz

Greetings,

A bug was recently discovered with Cloudflare, which Kraken and many other websites use for DoS protection and other services. Due to the nature of the bug, we recommend as a precaution that you change your Kraken security credentials:

Change your password
Change your two-factor authentication (remove and re-enable it)
Clients who use API keys should generate a new set of keys

You should similarly change your security credentials for other websites that use Cloudflare (see link below for a list of possibly affected sites). If you are using the same password for multiple sites, you should change this immediately so that you have a unique password for each site. And you should enable two-factor authentication for every site that supports it.

The Cloudflare bug has now been fixed, but it caused sensitive data like passwords to be leaked during a very small percentage of HTTP requests. The peak period of leakage is thought to have occurred between Feb 13 and Feb 18 when about 0.00003% of HTTP requests were affected. Although the rate of leakage was low, the information that might have been leaked could be very sensitive, so it’s important that you take appropriate precautions to protect yourself.

The problem is thought to have only started 6 months ago and 2FA or API keys generated before that time are probably not affected, but we recommend changing them anyway because the bug existed for years.

Here are some links for further reading on the Cloudflare bug:

TechCrunch article: https://techcrunch.com/2017/02/23/major-cloudflare-bug-leaked-sensitive-data-from-customers-websites/
List of sites possibly affected by the bug: https://github.com/pirate/sites-using-cloudflare/blob/master/README.md

If you have any questions or concerns in response to this email, please contact Kraken support at: https://support.kraken.com/hc/requests/new

Thank you for choosing Kraken, the trusted and secure digital assets exchange.

The Kraken Team