Back in Business

Those dirty rotten hackers

Just when I was starting to get some momentum with my blogging, misfortune struck. My first indication that something wrong was that I couldn’t ssh into my Linode instance. It was the first time I had tried since the day I launched it. “That’s odd, I thought. Nothing’s changed so it should be working.”

At this point I would have been completely out of luck if it weren’t for my Ajenti installation. Once I logged into Ajenti I was able to identify the problem. My authorized_keys file in my normal user’s home directory had bad permissions. Apparently they had come to belong to www-data:www-data and had permissions of 0755. “That’s not good. But maybe not out of the realm of normal stupidity on my part somehow”, I reasoned. Once I rectified the permissions, I found that I could ssh into my machine as my normal user, but I could not use sudo.

“Uh oh”. Back to Ajenti.

At this point I found out that my root user could not ssh in either. A quick look at the permissions told me that the same thing that had happened to my user’s authorized_keys file had happened to not only my root authorized_keys file, but it had happened to my /root folder… and all of the folders next to it… and all of the files and folders in them.

Cold Sweat.

Now that I was sitting there with my root login restored, staring at the output of ls -l /, I noticed something that made the whole story seem to come together. In addition to the normal /usr, /var, /etc... files, my root directory was populated with WordPress files. That was definitely not right. Somehow, it would seem, an attacker broke free of the docker container, probably using a vulnerability in WordPress. This seemed possible to me because I did have a directory on my host machine mounted onto the Docker container as wp-content/. wp-content/ is by far the most lenient of WordPress directories with regard to permissions and, really, I know almost nothing about the intricacies of WordPress vulnerability. “Well”, I thought, “Time to shut it down and start over”.

Nice try, guys

Just before I nuked my system to try to start over again I figured I’d try to get a snapshot of the database so I could start my blog again without missing a beat. This gave me some trouble as I didn’t have my MySql Docker container set up right. Out of curiosity, I checked the wp-config.php file to see if anything was amiss. Sure enough, DB_HOST was set to an IP I had never seen before: 172.17.0.2. For me that sealed it. An attacker had somehow managed to shim a fresh but identical installation of WordPress onto my server, but with at least one important difference: It wrote to their database. Clever girl. Luckily I had caught it in time…

Starting to rebuild

Shortly after the dust settled I set about rebuilding what was lost. Once I relaunched my Linode instance and got a few basic things installed and configured, I started in on relaunching the blog from scratch. This time I boned up more on Docker and WordPress security. One important change I made to my setup was adding some more sane permission defaults to my WordPress Docker image. I probably have quite a bit more work to do in this regard, but I think for now it’s probably at a roughly average level of security.

The Culprit Revisited

Part of launching my blog with Docker depends upon being able to mount a host volume onto the Docker container. The command I use to do that looks like this:

docker run --name some-wordpress --link some-mysql:mysql -v /path/to/content:/var/www/html/wp-content -d wordpress.

In this case, my /path/to/content was ./wp-content. Getting this to work in practice, however did not go so well. Time after time, I would fire up my Docker container with variations on that command and, sure enough, WordPress was running, but it was using a different wp-content directory. Neither my intended content directory, nor the one I accidentally mounted from my wordpress repository were being mounted to a location Docker was using, so therefore, it must have been using an internal one. That was my thinking until somehow I noticed an odd irregularity in my output of ls -l /:

drwxr-xr-x 2 root root 4096 Jul 1 14:33 bin
drwxr-xr-x 2 root root 4096 Apr 20 14:09 boot
drwxr-xr-x 12 root root 13780 Jul 2 05:00 dev
drwxr-xr-x 85 root root 4096 Jul 2 18:40 etc
drwxr-xr-x 3 root root 4096 Jul 1 19:08 home
drwxr-xr-x 18 root root 4096 Apr 20 13:52 lib
drwxr-xr-x 2 root root 4096 Apr 20 13:19 lib64
drwx------ 2 root root 16384 Apr 20 13:19 lost+found
drwxr-xr-x 3 root root 4096 Apr 20 13:20 media
drwxr-xr-x 2 root root 4096 Apr 10 22:12 mnt
drwxr-xr-x 2 root root 4096 Apr 16 21:02 opt
dr-xr-xr-x 113 root root 0 Jul 1 22:15 proc
drwx------ 4 root root 4096 Jul 1 19:17 root
drwxr-xr-x 15 root root 620 Jul 2 18:52 run
drwxr-xr-x 2 root root 12288 Jul 1 14:33 sbin
drwxr-xr-x 2 root root 4096 Apr 16 21:02 srv
dr-xr-xr-x 13 root root 0 Jul 1 23:11 sys
drwxrwxrwt 2 root root 4096 Jul 2 19:50 tmp
drwxr-xr-x 10 root root 4096 Apr 20 13:19 usr
drwxr-xr-x 12 root root 4096 Apr 20 13:23 var
drwxr-xr-x 4 www-data www-data 4096 Jul 2 19:49 wp-content

Notice anything unusual? The last directory listed is wp-content. Not only that, it contained all of the uploaded changes made to the running WordPress container. There was no question that this had somehow been my own doing, so I went back to the command that launched Docker:

sudo docker run --name some-wordpress --link some-mysql:mysql -v ./wp-content:/var/www/html/wp-content -d wordpress.

Was it possible that this relative path was ruining everything? To test this theory I deleted /wp-content and restarted my Docker container, this time with a full path specified. Guess what. It worked. So, hypothetically speaking, of course… if that one directory was copied erroneously to / through one careless command, what would happen if I accidentally were to have done that with not just the wp-content folder… but the whole wordpress codebase?

That’s right.

I was the idiot who “hacked” my blog in the first place. Suspecting foul play, I nuked the whole thing. I really did it. I’m a maniac. I blew it up. But what of the mysterious IP address to which the attackers had trained my blog to write? Well luckily, you can find the filesystem for any Docker image in the directory /var/lib/docker/containers/{container id}. Looking at my currently running WordPress wp-config I found that, sure enough, 172.17.0.2 is an internally used IP by which my WordPress Docker container connects to my MySql Docker container. Silly me.

Epilogue

Of course the actual command I probably used to break everything in the first place has been lost, buried in the sands of time. But this theory is the one that seems the most probable to me. The takeaway for me is twofold:

1. Be more careful
2. Have backups ready to go at a moment’s notice

In a future post I will describe my new setup which applies what I have learned. Until then I think I’ll settle into a big chair with a glass of scotch and reflect on my mistakes. Be careful out there, you guys. It’s a dangerous world.