Recovering from a hacked wordpress website

It was a dark and stormy night ...

The infection bootstrap file

It was late last night, and I wanted to make a slight change to one of my websites before heading to bed.  But when I brought up the home page of my site, I was startled when the normal front page was replaced by a glaring red Google warning that my site was serving up malware!

So much for going to bed anytime soon.  I've had this happen to me before, years ago: a long stagnant and nearly forgotten web site of mine used a third party plug-in which got hacked and pushed me onto the Google blacklist.  That was no fun to undo, and I wanted to clean up the current mess quickly enough to avoid being branded a malware site.

As a bit of background, the website in question runs on Dreamhost.com's hosting service and is built with Wordpress.  I've been a Dreamhost customer for about 10 years now and, while they've probably had more than their fair share of problems in the past, the combination of features, flexibility and price have kept me a reasonably loyal customer.

I SSH'd into my account to look around.  I discovered that the hack, as far as hacks go, was pretty simple.  Just about every PHP file (as far as I know, every single PHP file) on my account had one line of script added to it.  This script redirected people's browsers to a malware site.  That was interesting, because it meant that somebody was able to sweep through every file I owned and rewrite it.  It wasn't a site-specific attack -- all the sites I hosted, even a bunch of dead ones, were affected.  Somebody was able to either (a) run code on my account (which seemed most likely case -- that's how I'd do it), perhaps by having the viral payload in each PHP file try to infect other files on my account or, (b) had some FTP based program that kept running through and fetching files to infect them (seems unlikely and harder to implement).

As an aside, can I just say how much I hate PHP?  It's too-stupid to see a variable without a superfluous dollar sign.  It has a bastardized object model.  And it's got a security model that says as long as the file ends in .php  it gets run as code.  Code signing? Meta files to separate executable content from data? Nope.  If you can figure out how to upload a rogue .php file to a web site, you can run it with full permissions on the server.  That's just a constant invitation to disaster.  Like the one that happened to me.

Anyway, there I was, staring at a massive corrupting of all my web sites that used PHP.  The only thing that appeared to save me is the infection was additive rather than replacing code -- it just stuck a line of code at the beginning of each PHP file:

What got put in front of every PHP file

Still, I had two big tasks ahead: (1) stop any future damage and (2) remove all the current damage.  Mind you, it was not phrased like that in my mind.  It was more like "oh shit I got to fix this fast and figure out how to stop it from happening again." Neither turned out to be real fast, unfortunately.

Let me start with the second task, undoing the damage.  The damage consists of prepending the PHP files with an encoded script, most often in base-64 but sometimes just using hex escapes to obscure the evil logic.  I'm not sure why they bothered, though, other than to look cool. It sure did stand out when I looked: a more stealthy injection would be harder to find intermingled with all the other code in the PHP files.  But the consistent nature of the hack made it easy to spot.  So easy, in fact, somebody has written a fixit script to remove the changes.  So backing out the changes was no problem. (note that it removed the infection from my files, but did not find and remove the additional files placed in my web site.  Read below about those).

The first task was the real challenge and what took all the time.  I removed all the hacks to my files, but within a half hour they were all back again.  This meant that the infection process was still underway or had been retriggered. Not pleasant.

The term "fog of war" come to mind describing my state.  Somehow, I was able to uninflect files only to see them re-infected, but I had no idea how it was being done.  The following possibilities seemed to exist:

• Someone still able to log into my account
• Someone had super user access to the server
• Wordpress had some sort of flaw in it that allowed people to upload or modify PHP files
• Wordpress had some sort of flaw that allowed people to modify the database where they could insert code there (that would then infect the files)
• I missed some infected file that was being run and reinfected my sites
• Something else?? Some other hack that I had closed but leaked a user id or password to the server or database??

Immediately I took all my web sites off line -- I simply moved the contents of the websites into a separate directory and put a place holder index.html file into the root directory of each web site.  That bought me time.

I started by thinking it might be some sort of wordpress exploit.  I had turned on allowing people to register user ids in one of my blogs shortly before the mess had happened, so I guessed it might be some sort of SQL injection problem.  I proceeded to export my posts from wordpress, and then did a scorched earth reinstall.  I then re-uploaded my posts, rebuilt my customizations to the templates and put the web site back online.

And a half hour later, it had been hacked again.  Ok, that wasn't it.  Maybe an initial exploit opened the door for a subsequent one.  Or maybe it's that my hosting provider, Dreamhost, had been penetrated themselves and my assailant had root access. Or? It was still foggy.

To close the door on the first one, I went through a round of password resetting.  I changed the FTP/SSH password.  I changed the database password.  And then I cleaned the hacked PHP files and reopened the web site.

Again, before too long, the PHP files were getting hacked yet again.  This was frustrating.

I was starting to think that my hosting provider was hacked and there was nothing I could do. But I tried one more thing in desperation.  If I had my own server (say Amazon EC2), I would be able to exert a lot of control over the virtual machine -- reboot it, for example.  But with a shared hosting provider, I cannot.  But there's a small, almost hidden command in the control panel for Dreamhost websites, which terminates all existing FTP connections: what if someone had kept open an FTP connection? It seemed crazy, but I had all connections terminated.  I also went into my sites and had all existing cookies invalidated, so someone who had a login to my web sites would be forced off.  (If you're a wordpress user, look at the file wp-config.php and the section about changing the salts).

After that, I went through another round of changing passwords from the database on down.  I'm a Mac person, and the program 1password is invaluable in generating and maintaining a set of secure passwords.

I finally finished up very late last night, and today it appears that it is holding.

But there's one more thing ...

This morning, the hosting provider, Dreamhost, provided me a list of potentially hacked files in my account.  There were a whole bunch of other infected files I had missed, but whose infection patterns looked different. The first set -- the set I had seen and cleaned -- started with a PHP eval statement, but the second set used a bunch of hex escapes (see the first illustration in this entry).  My searches using grep didn't look for them, and the fixit script didn't see them either.  So I just deleted them all.  They were not infected files, they were the starter files to initiate an infection.  (They were all named either r.php, rr.php, or footlib.php -- the last of which was a small boot strap program that allowed someone to peruse the files in my account via the web.  It appeared to be the source of the initial infection).

I'm going to stay on top of my web sites for a while until I'm comfortable the threat has passed.  I'm not sure how the infection got in, so I'm not sure what to look out for.  Dreamhost had been hacked into itself in January, and I'm thinking that an infection script was put into one of my web sites where it laid dormant until last night.

All this is no fun.  If I wasn't familiar with Linux shell commands and PHP, I wouldn't know what to do. I guess host my blogs at Tumblr?  On the plus side, I've learned a lot about how wordpress works.  And did I mention how much I dislike PHP?