Keeping A Pulse On An Apache Web Server

Today I struggled with a customer's web server that kept spiking massive load averages. We're talking astronomical load averages, like 100 or 150. At that point, the box is so bogged down, it's not possible to debug what's going.

Because I couldn't tell what was happening when the box was actually in trouble, I decided that I'd set up a simple monitor that I could keep an eye on throughout the day. When I saw the box getting in trouble, I figured I could intervene before the server ground to a halt.

Here's what I did...

1. Setup mod_status, so that I could review live Apache status. See the sample below for what the status page looks like: While mod_status is plain text, it has to be one of the most expressive user interfaces out there. It strikes a remarkable balance of being both informative and high level.
2. Setup a command line to pull in the status every two seconds. Specifically, I ran the command:

    ( while sleep 2; date; 
       do rwebtool -G http://url/to/mod_status/output | \
       grep 'requests';
      done ) | tee -a ~/server-status
   

   The result is that every two seconds the following three lines are generated:

   Mon Nov 24 17:46:08 EST 2008
   <dt>32.9 requests/sec - 457.8 kB/second - 13.9 kB/request</dt>
   <dt>91 requests currently being processed, 40 idle workers</dt>
   

   This information is both logged on the screen and stored in the file server-status.

Throughout the day, I was able to keep an eye on the server - and as soon as I saw timeouts in the script, I new to visit Apache status page to see what was causing the server to get beaten up.

The script turned out to be quite useful, as I was able to capture the server just before it bit the dust. This then lead me to an unexpected solution.

Have I mentioned lately just how much I love Unix scripting?

Bonus: Other Commands To Monitor Server Usage

Here are some other commands I use to watch Unix servers:

  # show memory usage, refreshing every two seconds
  watch free -m

  # show a ascii art style load graph
  tload

  # watch the apache logs roll by
  tail -f /var/log/httpd/access_log

  # watch virtual memory usage
  vmstat  -S M 2