Chapter 2. Network Traffic Analysis

A network IDS is really just a network sniffer that compares the contents of packets of information traveling the wire to a catalog of signatures that indicate potential malicious activity. A sniffer is a device (formerly very expensive, special-built systems, but now a simple laptop) with a network card that watches traffic between computers and other network-capable devices. This device can do a number of things with this traffic: record, sort, or analyze it.

Because most network security and intrusion detection is based on identifying and interpreting packet data, it's important to understand how a packet is constructed and how it performs in real-world scenarios. In most cases, you can trust intrusion detection tools such as Snort and their alerts regarding suspicious packets, but there are times when the packet payload must be examined a person rather than a computer program. A careful analysis of a packet is sometimes required to determine if an alert is in fact a real alert or a red herring. Not knowing at least the basics of how computers use the network to communicate makes this task much harder, if not impossible.

This chapter starts with some level-setting discussions about how networks are used by systems to communicate using the TCP/IP suite of protocols. We'll cover the TCP/IP suite in general and concentrate on TCP in particular. While looking at TCP, we will break down the structure of an individual TCP packet, looking at the different options available. We will then examine the very important concept of the three-way handshake. This will be a quick survey of TCP/IP networking and is not meant to be a comprehensive education. The goal is to give you the tools you need to interpret what your IDS sensors are telling you.

One of the main tools used to capture and analyze network traffic is an open source tool called tcpdump. tcpdump is one of the most common tools for learning the basics of interpreting packets. It's easy to install on a number of platforms, freely available, runs on both Unix-based and Windows systems, and it's very flexible. I explain how to install and properly configure tcpdump and examine the basic usage of tcpdump as a teaching tool and a security application. I then look at ethereal, a graphical tool for examining network packets. ethereal has all the functions of very expensive commercial network analysis products and is an invaluable tool for a network administrator. The reason we start with the command-line-based tcpdump instead of the easy-to-use ethereal is to gain an understanding of what's going on under the hood. Since it is common to only have access to a remote command shell on a system, knowledge of the command-line tools at your disposal is vital. Once you become familiar with using a sniffer and discover the true value of watching your network at this level, ethereal will be at your side constantly. Finally, we will get to work and examine how systems establish and engage in conversations.