Apache symlink exploit heading towards being the worst in history

[brianoz]

I've been watching this one for a while and wanted to share about it as I think it's critical to being able to provide secure hosting going forward. At the time of writing, my guesstimate is that around 40% of hosts have been hit widely by this exploit. The really scary thing is the evidence I'm seeing that even very large hosts are being hit by it!

In case you haven't heard of it, here's a quick summary:

1. Hacker exploits one account on your server through conventional means - outdated WordPress or other CMS files, or stolen FTP password.
2. They then access /etc/passwd and generate a series of symlinks to other accounts of the form fred.txt --> /home/account/public_html/wp-config.php
3. They then visit the fed.txt file as a URL which shows them the wp-config.php file in plaintext, giving them the database username and password for the other account
4. If the user has re-used their cPanel password as a database password, they then have access to cPanel
5. They repeat this process until they have checked every single account on the server, looking for wp-config.php, configuration.php etc
6. Hey presto - a single account hack has been escalated to 70% of the sites on your server

Personally I think this is the most dangerous exploit method in the history of shared Apache webhosting since the old suphp/nobody exploit.

Also, the methodology here works via other methods other than symlink, which I prefer not to share as I don't want to give anyone ideas.

In a way, this isn't new, but what's happening now is that the scale of it and the automation attached to it has multiplied.

I've written this up at my blogs, including patches for Apache and some scripts to change modes on your .php files to keep them safe - see this URL: http://whmscripts.net/misc/2013/apache-symlink-security-issue-fixpatch/

If you're looking for a simpler intro to what this is all about, see this link: http://blog.whitedoggreenfrog.com/2...e-attacks-spread-through-webhosting-industry/

The whmscripts link will also take you through to an overly long discussion at cpanel and another patch being developed which eliminates the race condition that can be used to work around the other patch.

Perhaps it's just me, but I've found cPanel's attitude to this issue to be deeply disappointing. Of course, it's not their code, but they distribute an integrated system based on Apache and they're too lazy to apply a fix? When thousands of hosts are getting hacked? I don't get it - they've patched Apache for other things before. Anyway, rant mode off ... sorry

I can't encourage you strongly enough to apply both the patch and the permission workaround. There are many hundreds of thousands of hosts hacked by this worldwide at present, you don't want yourself or your customers to be a victim when 30 minutes work will save your server!

So, perhaps I'm being sensational? What have others found happening in the wild? Otherwise unexplainable hacks on servers is a good clue!

 

[TekStorm - James]

This isn't exactly anything new and is relatively easily guarded against if you don't have a need to allow it. Whether or not you choose to do so is your choice; it's not a cPanel problem. If you choose not to implement certain security policies on servers, and/or fail to harden certain things, that's a you-issue.

The symlink thing isn't really a bug that needs to be addressed/patched. It's an option that simply needs to be disabled if you don't have a reason for permitting it. If you decide to allow it, and not disable it, then it's functioning as it should.

 

[brianoz]

James, I wish that was the case! Unpatched Apache doesn't allow you to disable it. As you say, of course this is not new in itself; but the degree to which it is now being exploited is new.

Unpatched Apache allows you to set SymLinksIfOwnerMatch. However, it's trivial for the hacker to edit their .htaccess and remove that option and that's in fact what they are doing. Once they've done that, they can escalate a single account hack to include every account on the server. That's a big vulnerability and not just a small lack of functionality in my mind.

And of course, there's a race condition which means that SymLinksIfOwnerMatch can be entirely bypassed - and there's a reliable exploit for this. So, yes, I guess most people think this is an old bug so not important; I tend to think it's serious if it could lead to every account on a shared server being owned.

 

[TekStorm - James]

You can restrict what can and/or cannot be overridden in user-level .HTACCESS files, eh? Aside from that, there are different ways of restricting access between user accounts on the server; which, should really be in-place already anyway.

 

[brianoz]

You do realize that you can restrict what can and/or cannot be overridden in user-level .HTACCESS files, eh?

Yes, I do.

But there's a race condition which allows any restriction to be reliably worked around, and then server is owned by bad guy. There's an apache patch been mooted for this by a guy from bluehost, but it hasn't made it into Apache core yet. (The patch is, obviously enough, to check the file ownership AFTER the file is opened with fstat(2), rather than checking before and leaving a race condition window). Patch is given in the whmscripts article mentioned above.

 

[TekStorm - James]

Assuming that you're referring to the patch linked to from your own Web site, have you actually looked at its code and what it does?

It doesn't really do anything more than what I've already said that should be done on your own. All it really does is force certain configurables that should have been set anyway; assuming, you were wanting to restrict this.

There are reasons why one may not want to enable such restrictions, but if one does want to, they have the ability to do it without needing to use such a 'patch'.

As for race conditions, that's something different and that patch does nothing to mitigate/resolve them. All of that aside though, there are other ways to restrict access between accounts anyhow; more-or-less, making such race conditions a practical non-threat regarding symlinks between accounts.

Apache themselves reference such race conditions, and I'm aware of them. I'm not saying not to mitigate them if possible. However, I'm simply saying that there are systems available which should be in use already that make them an essential non-threat.


__ EDIT __
I see that you're referring to a different patch. I don't have time to have a look at that one at the moment, but will later. If it resolves the potential issues with race conditions, great. However, either way, there are ways that should have been in place to mitigate such attack vectors anyway.

 

[brianoz]

Hi James,

I don't mean to imply that there is no way to block the attacks other than the patch; of course, SymLinksIfOwnerMatch can be enabled globally and that does block much of the current attacks.

Nor do I question whether there are other means of stopping this exploit - in this case, setting file permissions on .php files to 600 or 640 are what I assume you're talking about.

This post is about educating hosting sysadmins in how to do what will keep their users safe. I'm simply making the point that, undefended, this is one of the worst holes in a long time because of it's potential to escalate to cover many accounts on a server. This has actually been done, repeatedlu, with from memory, around 200,000 accounts reported hacked by Badi alone last time I checked a few days back. And that's just what's been reported.

My point is - if you haven't done anything to protect yourself, do so now.

I did understand the patch, yes. It's value is in that it prevents an inexperienced admin from allowing any form of insecure FollowSymLinks, that is, it forces SymLinksIfOwnerMatch on. While this might not be an issue for a competent admin like yourself, it's definitely an issue for less competent admins and even some of the large companies who are getting hit.

I understand I used slightly sensational language in the title, but I do think this is one of the worst issues we've faced as an industry for a while.