xymon for disk performance

Enfield Cat's Blog: Arduino and other projects.

Introduction

Keeping a graph of I/O percentage utilisation is useful as often times of high utilisation may be at non-sociable hours. For example during overnight data-warehouse extracts and backups. The code below has been run on an Ubuntu 20.04 machine. But columns and column sequence may vary between CPU architecture, OS version etc. Generally percentage I/O utilisation will be the last column of output.

The client monitoring process

The code below is placed in a "iostat.sh" script in "usr/lib/xymon/client/ext" directory for a standard package installation of the client. Remember to set the code to executable before scheduling it: "chmod 755 /usr/lib/xymon/client/ext/iostat.sh" Note this script is written in a way that simply running it at the command line will display the xymon data to the terminal screen.

Note this is a bit of a hack that alters the default graph.cfg file that is installed / upgraded by the xymon package.

Check: /usr/lib/xymon/client/ext/vmware.sh

###########################################################################
#
# iostat monitor
#
###########################################################################
basedir=/home/xymon/client/ext

COLUMN="iostat"
export column
if [ -z "$XYMSRV" ] ; then
  SERVER=""
  XYMON="echo"
else
  SERVER="$XYMSRV"
fi
if [ -z "$MACHINE" ] ; then
  MACHINE=`/bin/uname -n`
fi
COLOR="green"

warnlim=80
critlim=98
interval=300

# sedstr=`ls -l /dev/mapper/ | awk '
sedstr=`ls -l /dev/mapper/ | awk '/dm-/ { if (NF==11) print $9 " " $11}' | sort -rk 2 | awk '
  /dm-/ {
    split ($NF, mapper, "/")
    piece = split ($(NF-1), filesys, "-")
    while (length(filesys[piece])<length(mapper[2])) filesys[piece] = sprintf ("%s_", filesys[piece])
    repeat = length(filesys[piece]) - length(mapper[2])
    if (repeat<0) repeat = 0
    printf (" -es/%s", mapper[2])
    for (n=0; n<repeat; n++) printf (".")
    printf ("/%s/g", filesys[piece])
  }
END { printf "\n" } '`

output=`iostat -x $interval 2 | awk '
/^Linux/ { print $0 }
/^avg-cpu:/ { chk++ }
{ if (chk>1) print $0 } ' | sed $sedstr | awk -v warnlim="$warnlim" -v critlim="$critlim" '
{
if (state > 0 && NF > 5) {
  if ($NF >= critlim) printf ("&red %s\n", $0)
  else if ($NF >= warnlim) printf ("&yellow %s\n", $0)
  else printf ("&green %s\n", $0)
  }
else printf ("   %s\n", $0)
}
/Device:/ { state++ }
' `
state=`echo "$output" | egrep -c "^&red|^&yellow"`
if [ $state -gt 0 ] ; then
  COLOR="blue"
fi

chk=`dpkg -l | grep -c sysstat`
if [ $chk -lt 1 ] ; then
  output="&blue sysstat package is not installed on this server."
  COLOR=blue
fi


$XYMON $XYMSRV "status $MACHINE.$COLUMN $COLOR `date` - iostat -x $interval 2
$output

Measured interval = $interval seconds
 Warning: Percent Util &gt; $warnlim
Critical: Percent Util &gt; $critlim
"
#
# Send through IO stat details
#
if [ $chk -gt 0 ] ; then
payload=`echo "$output" | awk '
/^\&/ {
printf ("[iostat.%s.rrd]\n", $2)
printf ("DS:r_req:GAUGE:600:0:U %s\n", $5)
printf ("DS:w_req:GAUGE:600:0:U %s\n", $6)
printf ("DS:util:GAUGE:600:0:U %s\n", $NF)
} '`

$XYMON $XYMSRV "data $MACHINE.trends
$payload
"
fi

Scheduling: /var/run/xymon/clientlaunch-include.cfg

Add these lines in the client machine's xymon schedule: /var/run/xymon/clientlaunch-include.cfg

The client process log files should be available in /var/log/xymon/iostat.log

[iostat]
        ENVFILE $XYMONCLIENTHOME/etc/xymonclient.cfg
        CMD $XYMONCLIENTHOME/ext/iostat.sh
        LOGFILE $XYMONCLIENTLOGS/iostat.log
        INTERVAL 5m

Server side changes

In order to get the iostat data to be included in a graph, some server side changes need to be made. Since there is already a graph definition in /etc/xymon/graphs.cfg we just need to tweak this to match the graph definition created above. Edit /etc/xymon/graphs.cfg, find the "[iostat]" section and make it look like this: 

[iostat]
        TITLE I/O Utilization
        YAXIS Pct Util
        FNPATTERN ^iostat.(.+).rrd
        DEF:p@RRDIDX@B=@RRDFN@:util:AVERAGE
        DEF:p@RRDIDX@R=@RRDFN@:r_req:AVERAGE
        DEF:p@RRDIDX@W=@RRDFN@:w_req:AVERAGE
        LINE1:p@RRDIDX@B#@COLOR@:@RRDPARAM@
        COMMENT:\n
        GPRINT:p@RRDIDX@B:LAST:   pct util\: %6.1lf (cur)
        GPRINT:p@RRDIDX@B:MAX: \: %6.1lf (max)
        GPRINT:p@RRDIDX@B:MIN: \: %6.1lf (min)
        GPRINT:p@RRDIDX@B:AVERAGE: \: %6.1lf (avg)\n
        GPRINT:p@RRDIDX@R:LAST: read req/s\: %6.1lf (cur)
        GPRINT:p@RRDIDX@R:MAX: \: %6.1lf (max)
        GPRINT:p@RRDIDX@R:MIN: \: %6.1lf (min)
        GPRINT:p@RRDIDX@R:AVERAGE: \: %6.1lf (avg)\n
        GPRINT:p@RRDIDX@W:LAST:write req/s\: %6.1lf (cur)
        GPRINT:p@RRDIDX@W:MAX: \: %6.1lf (max)
        GPRINT:p@RRDIDX@W:MIN: \: %6.1lf (min)
        GPRINT:p@RRDIDX@W:AVERAGE: \: %6.1lf (avg)\n
        COMMENT:\n


Thank you for visiting camelthorn.cloudHome