Weird request in my server log

IShotJR

new

Posts: 1

« Reply #15 on: November 26, 2007, 07:14:50 am »

I'm interested in finding out more about the type of request you're getting from our server.

I see that it keeps sending you the request over and over. Can you post the number of deny's as of Nov 23 to now?

Thanks,


	Report Spam Logged

madopal

Apprentice

Posts: 16

Re: Weird request in my server log

« Reply #16 on: November 26, 2007, 11:16:21 am »

It stopped on Saturday some time after I finally shut its opening down. Here's what what happening.

Apparently, there's a semi-known vulnerability in that apache, configured a certain way, will allow blind CONNECT and POST requests to the root of the server. What happens then is that the request will basically use POST as a defacto relay for spam. After looking into it more and more, I began checking my web logs. I started to see things like this:

Code:

199.8.89.120 - - [23/Nov/2007:21:33:11 -0600] "POST http://lti-mail01.ltinetworks.com:25/ HTTP/1.0" 400 317
199.8.89.120 - - [23/Nov/2007:21:33:11 -0600] "CONNECT http://lti-mail01.ltinetworks.com:25 HTTP/1.0" 400 305

Those two were of a reject (you can tell by the 400 after). The two numbers after the request are 1) the return code, and 2) the bytes transfered. So, these were the last failed attempts. From what I can gather, I relayed about 23 spam between Nov 1 (one of the first probes) and Friday when the attack started in earnest. Most of them were on Friday before I figured out what was happening.

The spurious GET requests that led me to this were the bot looking for a web connection to try. Since Ruby and other new servers are running on higher ports (I believe Ruby's default is 5000), the bots scan the higher ports looking for any webservers. They fire off a GET. If they get a response, they try the POST/CONNECT pair with the open mail relay (notice the request for port 25).

It's a NASTY exploit, because apache doesn't log the data in the POST usually, just the size. And because POST is such a common request, unless you're watching your web logs with a fine toothed comb, you're not going to see the request coming in about every 25 minutes. It's going to get hidden in the normal web log traffic. The only reason I saw it was that they hit my SSH port, which was ABOVE port 80 (not normal at all). If I hadn't seen it, I'd never have found this. That original request I got was a probe, and they forwarded a message. After I filtered through, I was added to the relay list, and the traffic started in full the day after Thanksgiving.

I verified this by telnet'ing into my server and manually adding a POST. It happily sat there waiting for more data. That's when I started looking into how to block it. Anyway, I shut down server, blocked some POST requests, and installed mod_security. Not sure which did it, but after that, the POST requests were returning 400 instead of 200. The bot kept trying for about 4 more hours (and came from 67.53.100.90 a few times, too...system is still up with ports open...rr.com DSL customer, I think), but finally gave up. I haven't seen a peep since.

Funny thing...nothing changed on my server. If this has been a vulnerability, I've had it for YEARS. I dunno if it's a default install from Fedora or what. I'm going to keep looking into what really causes it and see who I might let know about it.


	Report Spam Logged

Darksat

Administrator
Master

Posts: 3303

Re: Weird request in my server log

« Reply #17 on: November 27, 2007, 07:35:22 pm »

Well a quick look at header data shows that its another machine running the dot net framework.

Code:

HTTP/1.1 302 Found
Connection: close
Date: Wed, 28 Nov 2007 01:13:50 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
Location: http://199.8.89.120/Reports/Default.aspx
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Length: 157

Another .Net system
People should learn that a multi layer server technology built by monkeys will never be secure, especially with the translation layer allowing you to inject a load of different things into the server.
However this looks a lot more sophisticated.
Hacking/cracking an IIS machine has never been that hard, plus dot net has a load of holes, most of which are still being discovered. even if this one has v2.0, however the uni system looks fairly secure (for a university) and this hack is doing an IIS to Apache which suggests someone with a fair bit of skill.
The fact that they are scanning high ports, targeting apache, and the way this is put together leads me to believe that you are looking at someone who really knows how to take appart multiple systems.
I hope you have an IDS system like SNORT in place, not that it will do much use unless you know how to impliment it properly.
I would also consider remapping as many of your services running on your server to non standard locations, and if you can check if any other IP numbers in your range have been getting probed recently.
Install something like tripwire on your system just in case and I would start looking at all the activities on your mail server.
Also make regular backups and block the entire university range from accessing your system at all.
If someone with this level of skill decides to have a proper go at your system you would want to be well prepared.


	Report Spam Logged

madopal

Apprentice

Posts: 16

Re: Weird request in my server log

« Reply #18 on: November 27, 2007, 11:28:33 pm »

I'm running OSSEC, which is how I found out about the probes in the first place. It's got logging, checksum'ing, the works. It seems like it's very similar to Tripwire. I'll check if I can run both.

The bigger thing that bothers me is that this seems to be a common Apache config on Fedora (at least). I'm trying to suss out how the vulnerability got there. The timing of the attack (day after Thanksgiving) coupled with the frequency (once every 25 minutes) shows that whoever did this is patient, clever, and trying their hardest not be seen. Not the usual behaviors for spammers. Also, since you have to hack Apache to see what data was being sent with a POST, I have no idea what was being sent.

I'm sufficiently protected on the Apache side now, but I'm going to look into more sophisticated log analysis. All my other ports are (and have been) secure. If I hadn't been paranoid up to this point, I'd never have seen it.

I'll keep trying to poke at Apache/Fedora folk to find out why a blind POST to the webroot would allow this. It's not like they were using a php/cgi that was lying around. This is a default capability in Apache that seems to be enabled quite a bit.