Timing Reads

Using hdparm, you can time cache reads (-T) and device reads (-t).

• -T option-This option essentially checks the how quickly data can be passed between the processor, memory, and disk cache.

• -t option-This option reads continuously from the hard disk without reading previously cached data. This gives an indication of the speed at which Linux can read sequential data from the disk.

Here's an example of hdparm-Tt:

   # hdparm -Tt /dev/hda
   /dev/hda:
   Timing buffer-cache reads: 1476 MB in 2.00 seconds = 740.00 MB/sec
   Timing buffered disk reads: 164 MB in 3.01 seconds = 35.18 MB/sec

The output just shown is from a Western Digital Caviar 80MB hard drive (WDC WD800JB). The first line of output shows the number of MB of data (1,476MB) that can be read from cache in about 2 seconds, without touching the hard disk at all. The second line shows the number of MB of data (164MB) that can be read from the hard disk directly in about 3 seconds.

By way of comparison, I ran the same command on a Quantum Bigfoot CY6480A 6GB hard drive:

   # hdparm -Tt /dev/hda
   /dev/hda:
   Timing buffer-cache reads: 128 MB in 2.01 seconds = 63.56 MB/sec
   Timing buffered disk reads: 20 MB in 3.31 seconds =  6.04 MB/sec

You can see that the reads are considerably slower from this circa 1996 hard drive (not a bad command to try if you are looking to buy a used computer).

Here are a few tips for benchmarking with hdparm-Tt to get the best results:

• Have as few processes as possible active when you do the benchmark. (You can run the top command to see active processes. Type P (which is the default) to sort by CPU usage. Then you can kill any very active processes that you don't need.)

  Caution

  Don't just randomly go killing processes, even processes that are consuming a lot of CPU time. A better idea might be to go into single-user mode, since that should provide you with a system with very little activity.

• Run it a few times, to be sure a momentary hiccup doesn't skew your results.

• Have some free memory available. (If you are using top, press M to sort by memory usage and kill any process that is not required to free some memory.)

Timing Writes

To test options that change how writing is done to a disk (such as write-caching), you can simply time a large write operation with the options on or off. Here is an example where I used the time command to see how long it would take to copy a large file (in this case, a 637MB iso image) from one partition to another:

   # time cp severn-i386-disc1.iso /root
   real     0m34.295s
   user     0m0.110s
   sys      0m7.860s

The example just shown copies the iso image from the current directory to the /root directory. The time command shows that the operation took 34.295 seconds in real time. The user and sys lines show the amount of CPU time that was consumed by user-space calls and from kernel-space calls, respectively. Next, I turned off the write-caching feature (hdparm -W0), removed the iso file from /root, and ran the command again:

   # time cp severn-i386-discl.iso /root
   real  1m59.077s
   user  0m0.60s
   sys   0m8.900s

You can see that the copy took about four times as long (1 minute, 59.077 seconds) to complete. So, if you are willing to risk possible data loss in case power goes out, you can see that the time command shows significant improvement in completing a large copy operation with write-caching on in this case. (Of course, if you are that worried about data loss due to power going out, you should get a UPS.)

Instead of a plain copy, you can use the dd command to take a bit more control over the writes you test. With dd, you can precisely specify the block size and the number of blocks being copied. Using /dev/zero as the input file, you can copy as many "\0" characters to the output file as you like. Here's an example:

   # time dd if-/dev/zero of=/tmp/my-file bs=8k count=65536
   65536+0 records in
   65536+0 records out
   real    1m7.649s
   user    0m0.030s
   sys     0m3.780s

This created a 513MB /tmp/my_file. I can adjust the blocksize from 8k or the count from 65536 to test the disk performance based on different amounts and sizes of data.

Checking Reads, Writes, and CPU with Bonnie

Bonnie is a nice tool for checking performance of your hard disk, CPU, and other components in ways that emulate how the system is actually used. Using bonnie, you can check the actual system calls that are being made by the kernel to different devices as well as library calls to the basic libc library. The following are the types of performance activities you can check with bonnie:

• Programs that read and write single characters-Bonnie can test writes of single characters (putc call) and (getc call) to hard disks. The potential bottleneck for these operations is the time it takes to make libc library calls on your computer.

• Programs that read and write blocks of data-Bonnie can test writes (write call) and reads (read call) of blocks of data. Because performance of these activities may reflect buffering (that is, data that is in memory but not being currently read or written to hard disk), you might want to add some options to insure that you are testing disk performance and not just buffering. With the -y option to bonni e, buffered data is synced to hard disk after each write.

• Database applications-Bonnie can test activities that are similar to the way applications access data in a database. The block in/out rewrite test that bonnie does interacts with data on a hard disk by reading some data, changing several bytes, writing the data back to disk, and rereading the data. This is a kind of overall test of how well your system might handle database access.

• Busy system-By requesting more data than the system can handle (using the lseek call), bonnie can emulate how an overloaded system will perform. This tests how fast the hardware is at randomly accessing data from the hard disk.

The Bonnie project (www.textuality.com/bonnie) contains descriptions of bonnie and links where you can get the software (it's not included in any Red Hat distribution). I found a SuSE RPM version at www.garloff.de/kurt/linux/bonnie, which had a few nice enhancements. You can build from the source RPM you get there or use the SuSE i386 binary (which seems to work fine on Fedora and other Red Hat systems). You can also get that package from rpmfind.net (until they change the release number) by typing the following (all on one line):

   # wget rpmfind.net/linux/SuSE-Linux/i386/8.2/suse/i586/bonnie-1.4-
   124.i586.rpm

It is a good idea to run bonnie with amount of data larger than the default (100MB by default). Likewise, for a better test of hard disk performance, adding the -y option will work the disk harder by syncing all writes to disk. I ran the following bonnie command with write-caching on (hdparm-W1/dev/hda) or you could drop -y if write-cashing is off:

   # bonnie -s 300 -y
   Bonnie 1.4: File './Bonnie.18177', size: 314572800, volumes: 1
   Writing with putc()...         done: 20386 kB/s 89.0 %CPU
   Rewriting...                   done: 15913 kB/s 25.1 %CPU
   Writing intelligently...       done: 34819 kB/s 31.8 %CPU
   Reading with getc()...         done: 20819 kB/s 95.6 %CPU
   Reading intelligently...       done: 36583 kB/s 31.3 %CPU
   Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
              ---Sequential Output (sync)----- ---Sequential Input- -Rnd Seek-
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- -04k (03)
   Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
   toy 1* 300 20386 89.0 34819 31.8 15913 25.1 20819 95.6 36583 31.3 257.2 1.7

With the command just shown, 300MB of data (- s 300) is used for each operation and all writes are synced to disk (-y). Assuming a system with little or no extra activity, the CPU numbers above show cases where character-at-a-time writes (putc) and reads (getc) proceed about as fast as the processor can handle (89.0% and 95.6%, respectively). The hard disk, rather than the CPU, is the bottleneck for block writing and reading (31.8% and 31.3%, respectively), as well as rewriting (25.1%).

After that, I turned off write cache (hdparm-W0/dev/hda) and ran bonnie again.

   # bonnie -s 300 -y
   Bonnie 1.4: File './Bonnie.18151', size: 314572800, volumes: 1
   Writing with putc()...         done: 5950 kB/s 26.0 %CPU
   Rewriting...                   done: 5070 kB/s 8.0 %CPU
   Writing intelligently...       done: 6179 kB/s 5.6 %CPU
   Reading with getc()...         done: 20820 kB/s 96.0 %CPU
   Reading intelligently...       done: 33795 kB/s 29.7 %CPU
   Seeker 1...Seeker 2...Seeker 3...start 'em...done...done ...done ...
   ---Sequential Output (sync)----- ---Sequential Input- -Rnd Seek-
             -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- -04k (03)
   Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
   toy 1* 300 5950  26.0 6179  5.6  5070   8.0 20820 96.0 33795 29.7 226.2 1.8

As you might guess, writing to disk in particular slowed down quite a lot. Writing operations were three to five times faster with write cache enabled. For more information on options to use with bonnie, see the bonnie man page or the bonnie README file (in /usr/share/doc/ packages/bonnie).

Turning on Write-Back Cache

With the write-back option on, a write operation can appear to be done after data is written to cache, but before it is actually moved to the hard disk. The driver can move on to get more data, and delays that come from slow seek times and disk revolutions can be made up over time.

Risks to enabling write-back caches include:

• Power failures-If someone kicks out the power cord or if there is a power surge, it is possible that data the driver has been told was written to hard disk is not yet written on the physical hard disk. This can lead to data corruption.

• Data flushed out of order-It's possible that a second data block can be flushed to disk before the first block arrives, which might also cause data corruption.

• Data might become inconsistent-In instances where data must always be consistent, write-back caching should be turned off. In RAID systems where write stripes can span multiple disks, it is more likely for inconsistencies to arise if write-back caching is on.

Most modern hard drives have write cashing enabled by default. In fact, in many cases, the driver will not let you disable it. However, with some older hard drives, you can risk data corruption by turning this feature on. In fact, the hdparm command labels this feature as DANGEROUS.

If this feature is not already on (and you have assessed the risk of data loss as acceptable), you can turn on write-caching using the following command:

   # hdparm -W1 /dev/hda
   /dev/hda:
   setting drive write-caching to 1 (on)

When I used the time command to time a large file copy (see the Timing Writes section), there was significant real-time improvements (about four times faster) with write-caching on (-W1).

Turning on Read-Ahead Cache

With the read-ahead cache option turned on (which is the case by default in most cases), the hard disk's driver will read ahead a specified amount of data with the assumption that the read action will need more data. If the feature is turned on, you should see the following near the end of hdparm -I output:

   Commands/features:
     Enabled Supported:
        *    Look-ahead

You use the -A option to turn on (1) or off (0) the read-ahead feature. The following shows the feature being turned on:

   # hdparm -A1 /dev/hda
   /dev/hda:
   setting drive read-lookahead to 1 (on)

With read-ahead on, you can try different values to set how far ahead data should be read. You do that with the -a option. For example, to set the read-ahead value to eight sectors (4KB) you would type

   # hdparm -a8 /dev/hda
   /dev/hda:
    setting fs readahead to 8
    readahead    = 8 (on)

With a slow, older hard disk, it can help to have a higher read-ahead value (to a point). The following examples show how read performance differed when I changed the read-ahead values with my old Quantum Bigfoot hard drive (described earlier).

The following shows read performance with read ahead turned off (-A0). I had similar results when I set read ahead to 8 (-a8).

# hdparm -a8 /dev/hda
# hdparm -Tt /dev/hda
/dev/hda:
 Timing buffer-cache reads: 128 MB in 2.00 seconds = 63.98 MB/sec
 Timing buffered disk reads: 10 MB in 3.60 seconds = 2.77 MB/sec

The next example shows how performance improved with read ahead set to 128 (I had similar results on setting it to 256):

# hdparm -a128 /dev/hda
# hdparm -Tt /dev/hda
/dev/hda:
 Timing buffer-cache reads:  128 MB in 2.01 seconds = 63.69 MB/sec
 Timing buffered disk reads:  22 MB in 3.24 seconds = 6.79 MB/sec

Setting read ahead to 64 resulted in improved performance, although not quite as much as setting it to 128 or 256:

# hdparm -a64 /dev/hda
# hdparm -Tt /dev/hda
/dev/hda:
 Timing buffer-cache reads:   128 MB in 2.01 seconds = 63.56 MB/sec
 Timing buffered disk reads:  20 MB in 3.25 seconds = 6.16 MB/sec