Apache DOS attack – CLOSE_WAIT

Our servers recently were crashed due to a large number connections to web (apache) services, we didn’t see anything abnormal on the web. The server will crash when there are over 150 connections, these connections could just come from one IP. After a long investigation, we realize that the attacker attack us by sending a lot of connections, but somehow they put all the TCP status in CLOSE_WAIT status, apache can’t release this.

The solution is , we write this script to kill all httpd process having CLOSE_WAIT status. This seems fixing our problems.

<?
//Some apache process are in close_wait
$cmd="netstat -ntp | grep CLOSE_WAIT | grep httpd"; //replace httpd with apache or something else
$process_array=array("httpd");
$hostname=trim(shell_exec("hostname -s"));
if($hostname=='ubuntu')
	$cmd="netstat -ntp ";
debug("Running $cmd");
$netstat=shell_exec($cmd);
$file_log="/tmp/kill_close_wait.txt";

$arr=explode("\n",$netstat);
$test=false;//if we are debugging- if we are debug - it will not killl
$log=false;
foreach($arr as $line)
{
	$line=preg_replace('/\s{2,}/',' ', $line);
	$arr2=explode(' ',$line);
	if(isset($arr2[6]) &&  $arr2[6]<>'-')
	{
		
		$arr3=explode('/',$arr2[6]);
		$process_id=$arr3[0];
		
		
		if($process_id==0)
			continue;
		$process_name=$arr3[1];
		if(!in_array($process_name,$process_array))	
			continue;
		
		debug("Found: $line: with process_id: $process_id");
		$cmd="kill $process_id";
		if(!$test)
		{
			
			debug("Kill $cmd");
			shell_exec("$cmd");
			$log=true;
		}else
		{
			debug("Test - not kill : $cmd");
		}
		
	}
	
}

if($log)
{
	$content="--------------------------------------------".date("Y-m-d H:i:s")."-------------------------------------------------------- \r\n $netstat";
	write_file($file_log,$content);

}

function debug($s)
{
	echo "\r\n $s";
}
function write_file($filename,$content)
{
	$fp = fopen($filename, 'a+');
	
	fwrite($fp, $content);
	fclose($fp);
}
?>