The tool chain of a sys admin should always be comprised of effective tools. Today we are introducing the package sysstat.

Sysstat is a collection of command line tools dedicated to providing the system administrator with a quick overview of the performance of the system. They work as front-ends to the Kernel and therefore can never provide more data than the Kernel itself gathers, although the interface is much more user-friendly than querying Kernel parameters manually.

iostat

iostat is the way to go if there are problems with the throughput of a disk, NFS storages or the CPUs. For example, if your system is behaving strangely, iostat can be used to identify I/O waits:

Linux 2.6.31-19-generic (mymachine)         04.03.2010      _x86_64_        (2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11,82    0,29    3,44    1,25    0,00   83,20

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               9,39       161,19       168,44    4264806    4456696

There are many options available for iostat but these are the most interesting, and they deal with specific outputs:

-d

Just show the hard disk data.

-c

Just show the CPU data.

-p

Show the I/PO data for each partition.

-n

Show the I/O data for the NFS partitions.

-x

Extended information for the hard disks.

-t $NUM1

Tells the program after how many seconds the result should be refreshed.

mpstat

mpstat is the next tool in the chain: it helps when analysing the CPU load. If you call it with no options, the default information will be shown, in the same way that the iostat results are.

Linux 2.6.31-19-generic (mymachine)         04.03.2010      _x86_64_        (2 CPU)

17:01:52     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
17:01:52     all   11,96    0,29    3,26    1,23    0,10    0,11    0,00    0,00   83,06

In contrast to iostat, you see the actual load on hard and software interrupts. The option -A extends this information further: for each processor, the statistics and interrupts per second are shown.

If you add an int $NUM after the command, the process runs without end and refreshes the output every $NUM seconds.

pidstat

pidstat concentrates on the processes itself: it shows a list of all processes. The option -C enables you to filter these by a given string:

Linux 2.6.31-19-generic (mymachine)         04.03.2010      _x86_64_        (2 CPU)

17:02:32          PID    %usr %system  %guest    %CPU   CPU  Command
17:02:32            1    0,00    0,00    0,00    0,00     1  init
17:02:32         2888    0,00    0,00    0,00    0,00     0  start_kdeinit
17:02:32         2889    0,00    0,00    0,00    0,00     0  kdeinit4

The additional option -d shows I/O statistics about the given processes, -p takes the PID as an argument to focus on known processes. Finally -r brings up an overview of the memory load.

Again, an int $NUM after the command lets the process run continuously, refreshing the output every $NUM seconds.

sar

All sysstat tools so far have had one flaw, only showing a snapshot of the current state and unable to look into the behaviour of the system in the past or during load time. Such information must be collected in the background, which is exactly what sar and its tools are all about: it collects the performance data of the system every ten minutes via cron job. If you call the tool with the default values you get a first impression:

Linux 2.6.31-19-generic (mymachine)         04.03.2010      _x86_64_        (2 CPU)

09:30:30          LINUX RESTART

09:35:02        CPU     %user     %nice   %system   %iowait    %steal     %idle
09:45:01        all     17,38      1,02      5,10      3,87      0,00     72,63
09:55:01        all     11,90      0,27      2,86      0,75      0,00     84,23
10:05:01        all     10,20      3,52      3,46      2,55      0,00     80,27
10:15:02        all     12,96      0,32      3,18      0,65      0,00     82,89
10:25:01        all      7,94      0,18      3,17      2,42      0,00     86,30
10:35:01        all     12,41      0,89      4,55      0,56      0,00     81,60
10:45:02        all      8,97      0,09      3,51      0,89      0,00     86,55

All possible information can be collected with sar -A although the amount of output will be too much for any screen size. There are too many options involved in decreasing the output with sar to cover here, but they are discussed in detail on the man page.