Thursday, December 15 was a day like any other until the afternoon. Then I got the notice from the Jetpack plugin for one of my WordPress sites that it was down, and couldn’t be reached.
This happens occasionally, so I wasn’t too upset. Pointing my browser to the site Home2Baja gave a Database connection error. Simple enough to fix.
So I fired up PuTTY, and logged in. I attempted to restart MySQL, the first line of fixing the issue. Weirdly, it restarted, then stopped automatically again.
What f*ckery is this?
So I restart the droplet (this is hosted on the incredibly awesome service Digital Ocean) and after a minute try to browse to the site again. Same issue.
Grrrrr.
So I once again fire up PuTTY, and log in. Now all sorts of bat-shittery is happening. I am getting BASH errors, not enough memory to fork. I can’t even log in, so I go to the Droplet service on DO, and log into the console.
… and the screen fills with Apache error codes.
A little Google-fu, and it appears that the site is getting hammered with XML-RPC requests, causing Apache to use all the memory, and essentially shutting down the droplet.
The problem was that I could power it off, and on, but before I could SSH in, the site was jacked with the cascade of XML-RPC requests.
Finally, I got in, and was able to apply a fix (also, documented well on the Digital Ocean support knowledge base), and got it back under control.
Now, I have Cloudflare running interference, so that in the future if/when I get hammered like this again, I can block it without being locked out of my own VPS.
A fun afternoon.
(Background: The “Home2Baja site is a website I created for a friend who is selling his home in San Felipe, B.C. We use Google Adwords to drive traffic to it, and it gets 30 – 50 hits a day. Clearly someone pointed their attack vector at it, and it was getting 4,000 xml-rpc queries a second. No wonder why my measly 1gb droplet was getting inundated. Yes, there is a firewall, a fairly restrictive firewall, but these queries come via HTTP, or port 80.)
