It stopped on Saturday some time after I finally shut its opening down. Here's what what happening.
Apparently, there's a semi-known vulnerability in that apache, configured a certain way, will allow blind CONNECT and POST requests to the root of the server. What happens then is that the request will basically use POST as a defacto relay for spam. After looking into it more and more, I began checking my web logs. I started to see things like this:
199.8.89.120 - - [23/Nov/2007:21:33:11 -0600] "POST http://lti-mail01.ltinetworks.com:25/ HTTP/1.0" 400 317
199.8.89.120 - - [23/Nov/2007:21:33:11 -0600] "CONNECT http://lti-mail01.ltinetworks.com:25 HTTP/1.0" 400 305
Those two were of a reject (you can tell by the 400 after). The two numbers after the request are 1) the return code, and 2) the bytes transfered. So, these were the last failed attempts. From what I can gather, I relayed about 23 spam between Nov 1 (one of the first probes) and Friday when the attack started in earnest. Most of them were on Friday before I figured out what was happening.
The spurious GET requests that led me to this were the bot looking for a web connection to try. Since Ruby and other new servers are running on higher ports (I believe Ruby's default is 5000), the bots scan the higher ports looking for any webservers. They fire off a GET. If they get a response, they try the POST/CONNECT pair with the open mail relay (notice the request for port 25).
It's a NASTY exploit, because apache doesn't log the data in the POST usually, just the size. And because POST is such a common request, unless you're watching your web logs with a fine toothed comb, you're not going to see the request coming in about every 25 minutes. It's going to get hidden in the normal web log traffic. The only reason I saw it was that they hit my SSH port, which was ABOVE port 80 (not normal at all). If I hadn't seen it, I'd never have found this. That original request I got was a probe, and they forwarded a message. After I filtered through, I was added to the relay list, and the traffic started in full the day after Thanksgiving.
I verified this by telnet'ing into my server and manually adding a POST. It happily sat there waiting for more data. That's when I started looking into how to block it. Anyway, I shut down server, blocked some POST requests, and installed mod_security. Not sure which did it, but after that, the POST requests were returning 400 instead of 200. The bot kept trying for about 4 more hours (and came from 67.53.100.90 a few times, too...system is still up with ports open...rr.com DSL customer, I think), but finally gave up. I haven't seen a peep since.
Funny thing...nothing changed on my server. If this has been a vulnerability, I've had it for YEARS. I dunno if it's a default install from Fedora or what. I'm going to keep looking into what really causes it and see who I might let know about it.
