Preventing DDoS attacks

By Blessen Cherian and Ben Okopnik

Note from the Editor

	C is as sphere as Earth. It's center is everywhere and circumference is
	nowhere and hence what you see in daylight is only one percent of what you
	can see in darkness.
	 -- cited from an article submitted to LG (anonymous)

With the agreement of the original author of this article, I've listed myself as co-author here, since I essentially rewrote the article that was submitted. Normally, proofing an article and adding some HTML structure is just part of the job here at LG; however, complete reformatting of idiom, recasting of nearly every paragraph, and updating the technical information is well beyond the scope of what is normal.

I hate to reject an article that has excellent technical merit almost as much as I want to avoid publishing one that would be unintelligible to many of our readers (particularly those for whom English is not their primary language, or who have difficulty parsing it for other reasons.) However, I also feel that doing what is essentially a major rewrite of an article should not go unrecognized. Since this is the first time I've ever explicitly taken credit for doing this kind of major reconstruction - and since my own vewpoint here could be tainted by the fact that it's my work that's involved - I want to solicit comments, ideas, and suggestions from you, our readers. Anyone want to volunteer as a stand-by co-author? Got an alien idiom-conversion ray that's been in your attic for the last hundred years? Do you see some other obvious solution I've missed? Bring'em on; the 'Talkback' link at the end of this article is your friend.
-- Ben Okopnik, Editor-in-Chief

---

Introduction

In this article, I will try to explain what DDoS is, and how it can be prevented or mitigated. Many of the servers in datacenters these days are Linux-based; hence, I'm going to discuss DDoS attack prevention and mitigation for Linux servers.

DDoS happens due to lack of security awareness, application, or skill on the part of the network/server owners or adminstrators. We often hear that a particular machine is under DDoS attack, or that the NOC has unplugged a given machine due to its participation in a DDoS attack. DDoS has become one of the common issues in our world. In some ways, DDoS is like a disease which doesn't have a countering antibiotic, and requires being very careful while dealing with it. Never take it lightly. In this article, I'll try to cover the steps/measures which will help us defend our machines from a DDoS attack - at least up to a certain extent.

What Is A DDoS Attack?

Simply stated, DDoS (Distributed Denial of Service) is an advanced version of the DoS (Denial of Service) attack. Much like DoS, DDoS also tries to block important services running on a server by flooding the destination server with packets. The specialty of DDoS is that the attacks do not come from a single network or host but from a number of different hosts or networks which have been previously compromised.

DDoS, like many other attack schemes, can be considered to consist of three participants; we can refer to these as the Master, the Slave, and the Victim. The Master is the initial source of the attack - i.e., the person/machine behind all this (sounds COOL, right?) The Slave is the host or network which was previously compromised by the Master, and the Victim is the target site/server under attack. The Master informs the Slave(s) to launch an attack on the victim's site/machine; since the attack comes from multiple sources at once (note that the Master is usually not involved in this phase), it is called a Distributed (or co-ordinated) attack.

How Do They Do It?

DDoS occurs in two phases. In the first phase, the owner of the Master host compromises vulnerable machines in different networks around the world and installs DDoS tools (i.e., programs that will perform the attack once they're triggered.) This is called the Intrusion phase. In the next phase, the Master sends out the triggering information to those compromised hosts, which usually includes the IP to be attacked (conversely, that IP could have been pre-programmed into the tools, and the attack could be time-triggered - e.g., the Code Red virus DDoS against the http://whitehouse.gov servers.) This is called the Attack phase.

What Allows Them To Do It?

The success of the Intrusion phase relies on the presence of vulnerable machines on an arbitrary network. Unfortunately, there's a very large number of naive computer owners and system administrators whose machines are largely unprotected, and thus this phase will be easily accomplished by the attacker in almost all cases.

Some of the factors that make the Slaves-to-be vulnerable are:

1. Vulnerable software/applications running on a machine or network.
2. Open/unprotected network configuration.
3. Hosts configured without taking security into account.
4. Absence of monitoring or data analysis.
5. No regular audit or software upgrades being conducted.

What Should We Do If We Are Under Attack?

If your host is one of the Slaves in a DDoS, you will most likely never even be aware of it - unless you carefully examine your logs and watch for untoward network activity. If, on the other hand, you're the Victim, the results will be dramatic and obvious.

Symptoms (Victim):

1. Programs run very slowly
2. Services (e.g., HTTP) fail at a high rate
3. Large number of connection requests from different networks
4. User complaints about slow (or no) site access
5. Machine shows a high CPU load

If you discover that you're under attack, follow these steps:

• Check if your CPU load is high and you a have large number of HTTP process running
  Check the load using the 'w' or the 'uptime' commands:

  Blessen@work >w
   12:00:36 up 1 day, 20:27, 5 users, load average: 0.70, 0.70, 0.57 
  

  Count the number of HTTP processes (it helps to know what your normal count is for comparison):

  [root@blessen root]# ps -aux|grep -i HTTP|wc -l 
  23 
  

• Determine the attacking network
  In a heavily-loaded server, the number of connection may be above 100 - but during a DDoS attack, the number will go even higher. That's when we need to find out, as quickly as possible, which networks are launching these attacks. In a DDoS attack, the individual slave machine doesn't have much importance; it is the network which matters the most, since an attacker could be using any or even all of the machines on a compromised network. Consequently, the network address is of crucial importance.

  Executing the following command will show the IPs arranged in order of established connections:

  bash# netstat -lpn|grep :80|awk '{print $5}'|sort
  

  For an average host, if you have more than 30 connections from a single IP, chances are that you're under attack. In normal operation, there is very rarely any reason for that many connection requests from a single IP. Identify these networks for later reporting, perhaps by using the 'whois' command.

  If more than 5 such hosts/IPs connect from the same network, that's a very clear sign of DDoS.

• Block the attacking network
  This can be done by using 'iptables' or 'apf':

  iptables -A INPUT -s <Source IP> -j DROP 
  

  If you're running 'apf', simply add these IPs to the '/etc/apf/deny_hosts.rules' file. Continue this elimination process until the attack on the machine is reduced (and hopefully, eventually stopped altogether.) As a follow-up measure, contact the datacenter/NOC responsible for that network to inform them of the compromised systems.

  As a longer-term strategy, once the immediate attack is over (or, if you're smart, you can do it right now :), install Portsentry (see the software listed at the end of this article.)

How can we prevent or defend ourselves from these attacks?

There is no complete or perfect solution to DDoS. The logic is simple: NO software or countermeasures can stand up to attacks from, say, 100 servers at once. All that can be done is to take preventive measures, and respond quickly and effectively when the attack takes place.

As it is often said, an ounce of prevention is better than a pound of cure - and this is very true in the case of DDoS. In the introduction, I had mentioned that DDoS often happens because of vulnerable software/applications running on a machine in a particular network. Attackers use those security holes to compromise the hosts and the servers and install the DDoS tools such as 'trin00'.

To prevent or mitigate future DDoS attacks, follow these steps:

• Create and implement a good security policy
• Set up a firewall which does ingress and egress filtering at the gateway (e.g., APF from http://www.rfxnetworks.com/apf.php)
• Use host-based intrusion detection on your gateway/hosts to alert you to port scans and break-in attempts (e.g., AIDE from http://freshmeat.net/projects/aide/)

To prevent your network from being used as a slave, follow these steps:

• Conduct regular audits on each host on the network to find installed DDoS tools and vulnerable applications.
• Use tools like Rkdet, Rootkit Hunter, or chkrootkit to find if a rootkit has been installed on your system.
• Perform a general security audit on your systems on a regular basis:
  • Keep your systems up to date to minimize software vulnerabilities (kernel and software upgrades)
  • Check for rootkits
  • Check logs for evidence of port sniffing, etc.
  • Check for hidden processes by comparing the output of 'ps' and 'lsof'.
  • Use auditing tools (i.e., Nessus, SAINT, or SARA)
  • Check system binaries with, e.g., Tripwire to see if they've been changed since your last snapshot
  • Check for open email relays
  • Check for malicious cron entries
  • Check /dev /tmp /var directories for odd files (i.e., '...', wrong permissions/ownership on device files, etc.)
  • Check whether backups are maintained
  • Check for unwanted users and groups (examine /etc/passwd)
  • Check for and disable any unneeded services
  • Check for SUID, SGID, and 'nouser' files on your system with the 'find' command
  • Check the system performance (memory and CPU usage); note the average levels
• Create a DSE (Dedicated Security Expert) team for your company.
• Enforce and implement security measures on all hosts in the network. The only hosts that should be allowed on your network are ones that have been vetted by your security admin or DSE (Dedicated Security Expert). All hosts on the network should be checked on a regular basis by your DSE team.
• Collect your network and host data and analyze them to see what kind of attacks are being run against your networks.
• Implement Sysctl-based protection. Enable the following in your '/etc/sysctl.conf':

  # Enable IP spoofing protection, turn on Source Address Verification 
  net.ipv4.conf.all.rp_filter = 1 
  
  # Enable TCP SYN Cookie Protection 
  net.ipv4.tcp_syncookies = 1 
  

  Conversely, you could add this code to your '/etc/rc.local':

  for f in /proc/sys/net/ipv4/{conf/*/rp_filter,tcp_syncookies}
  do
  	echo 1 > $f
  done 
  

• Install PortSentry to block scanning hosts.
• Add 'Mod_dosevasive' to your Apache installation. This is an Apache module which performs 'evasive' action in the event of an HTTP DDoS attack or brute force attack.
• Install the 'Mod_security' module. Since DDoS often targets HTTP (port 80), it's a good idea to have a filtering system for Apache; 'Mod_security' will analyze requests before passing them to the web server.
• Set up load balancing for your services. In some ways, this is the most powerful network-based defense against DDoS.
• Create awareness of security issues.

Conclusion

DDoS attacks can be mitigated at the target machine and prevented at the slave network by implementing proper security. My advice to each and every server and network owner is to implement effective security measures; since DDoS is a network-wide problem, preventing it is going to require everyone's help.

Talkback: Discuss this article with The Answer Gang

Blessen Cherian

---

My name is Blessen and I prefer people calling me Bless. I got interested in Linux when I joined the software firm, Poornam Info Vision Pvt Ltd also known as Bobcares. They gave me exposure to linux.

I am a B.Tech in Computer Science from the College of Engineering, Chengannur. I passed out in the year 2001 and got into the company that year. During my work, I was passionate with Linux security and I look forward to grow in that field.

My hobbies are browsing net, learning new technologies and helping others. In my free time I also develop open source softwares and one of them is a scaled down version of formmail. The project is called "Smart Mail" which is more secure than formmail.

Ben Okopnik

---

Ben is the Editor-in-Chief for Linux Gazette and a member of The Answer Gang.

Ben was born in Moscow, Russia in 1962. He became interested in electricity at the tender age of six, promptly demonstrated it by sticking a fork into a socket and starting a fire, and has been falling down technological mineshafts ever since. He has been working with computers since the Elder Days, when they had to be built by soldering parts onto printed circuit boards and programs had to fit into 4k of memory. He would gladly pay good money to any psychologist who can cure him of the recurrent nightmares.

His subsequent experiences include creating software in nearly a dozen languages, network and database maintenance during the approach of a hurricane, and writing articles for publications ranging from sailing magazines to technological journals. After a seven-year Atlantic/Caribbean cruise under sail and passages up and down the East coast of the US, he is currently anchored in St. Augustine, Florida. He works as a technical instructor for Sun Microsystems and a private Open Source consultant/Web developer. His current set of hobbies includes flying, yoga, martial arts, motorcycles, writing, and Roman history; his Palm Pilot is crammed full of alarms, many of which contain exclamation points.

He has been working with Linux since 1997, and credits it with his complete loss of interest in waging nuclear warfare on parts of the Pacific Northwest.