Cfiltl -- a filter for collectl

Cfiltl, a filter for Mark Seger's "collectl"

DESCRIPTION

Cfiltl enables one to apply a grammar to the plot output of collectl. If you have ever used "cfilt" on Tru64, cfiltl will seem very familiar.

GRAMMAR

Components

some common math functions are allowed: "int", "sin", "cos", "log", "sqrt", etc
arithmetic operations are allowed: +, -, *, /, %, **
parenthetical expressions
summing (see below)
normalization (see below)

The column-names ("tags") from collectl's output (with some minor adaptations) are used as placeholders for the values extracted from each record. One or more subsystem-expressions can be given to cfiltl. Multiple subsystem expressions can be provided for the same subsystem.

Subsystem-Expression

The format of a subsystem expression is (things between "[" and "]" are optional):

<3-char-unique-subsystem-name>[s|+][=item1,item2,item3,...]:<tag-expr1>:<tag-expr2>:...:<tag-exprN>

Tag-Expression

A <tag-expr> is an arithmetic expression containing the grammatical components listed above. Tag-expressions are evaluated once for each collectl record.

Some examples:

user simply extract the value in the user column for the subsystem
user+sys result is sum of the values in the user and sys columns for the subsystem
int((user+sys)/8) add user and sys values, divide by 8, and truncate to integer value

user		simply extract the value in the user column for the subsystem
user+sys		result is sum of the values in the user and sys columns for the subsystem
int((user+sys)/8)		add user and sys values, divide by 8, and truncate to integer value

Subsystem Selection

The first part of the subsystem-expression controls:

what subsystem is chosen
whether "summing" is enabled for the subsystem-expression
whether summary or detail subsystem is desired
for a detail subsystem, whether individual objects are selected

Some examples:

cpus select CPU SUMMARY subsystem. Summing is not possible and will result in error. No selection-criteria can be specified for a summary subsystem.
cpu CPU detail subsystem. No selection-criteria, so summing is enabled by default.
cpu=0,1 CPU detail subsystem. Each tag-expression will be evaluated for CPUs 0 and 1, individually.
cpu+=0,1 CPU detail subsystem. Each tag-expression will be evaluated for the sum of the values for CPUs 0 and 1.

cpus		select CPU SUMMARY subsystem. Summing is not possible and will result in error. No selection-criteria can be specified for a summary subsystem.
cpu		CPU detail subsystem. No selection-criteria, so summing is enabled by default.
cpu=0,1		CPU detail subsystem. Each tag-expression will be evaluated for CPUs 0 and 1, individually.
cpu+=0,1		CPU detail subsystem. Each tag-expression will be evaluated for the sum of the values for CPUs 0 and 1.

Normalization

Normalization means forcing data values into a particular range, by default between 0 and 100. This allows plotting of such data on the same graph with other data that either naturally falls into the same range, such as CPU load, which is always a percent, or data that has been similarly normalized.

There are two normalization notations:

"old-style" -- A hash ("#") can be appended to a "tag" (column-name), with an optional number immediately following. This notation can only be used to normalize a single tag. It's available for convenience.
"new-style" -- by using the pseudo-function "norm(<expression>[,<max>])". This allows normalization of an arbitrary expression, for example "norm(sqrt(rxkb+txkb))".

Normalization requires that causes cfiltl to read all records, store the relevant data in memory, and then calculate the "normalized" value for each record-value based on the highest overall value found in all collectl records.

Nested normalization ("norm(intrpt/norm(ctx))") is not allowed.

Some examples:

norm(rxkb+txkb)    normalize sum of received and transmitted KB (net) to the range 0-100
norm(rxkb+txkb,200)    normalize sum of received and transmitted KB (net) to the range 0-200
intrpt#    normalize cpu interrupt values to the range 0-100
intrpt#200 normalize to range 0-200

norm(rxkb+txkb)		normalize sum of received and transmitted KB (net) to the range 0-100
norm(rxkb+txkb,200)		normalize sum of received and transmitted KB (net) to the range 0-200
intrpt#		normalize cpu interrupt values to the range 0-100
intrpt#200		normalize to range 0-200

Summing

If a plus-sign is appended to a detail subsystem name, as in "cpu+", a tag in the tag-expression is replaced by the sum of all values in the corresponding column of collectl's output, or if a subset of objects in a subsystem has been selected, the sum for the selected objects. For example, to sum TxKB for "eth0" and "eth1":

net+=eth0,eth1:TxKB

Specifying a detail subsystem without selecting any objects implies summing.

See the comments in the script for more detail.

TAG NAMING

Tag-names are taken from collectl's plot-format column-name headers with the following restrictions/exceptions:

all column-names are converted to lower-case
if a column-name contains an arithmetic operator (%*/+-) (examples: "Wait%", "Intrpt/sec") the operator and everything following it is removed
a few column-names contain dashes: "L-Avg1", "L-Avg5", "L-Avg15". These are converted to "ldavg1", "ldavg5", and "ldavg15"

Thus:

Wait% becomes "wait"
Intrpt/sec becomes "intrpt"
proc/sec becomes "proc"

Wait%	becomes "wait"
Intrpt/sec	becomes "intrpt"
proc/sec	becomes "proc"

Otherwise, tags must correspond 1-to-1 to column-names. Thus, proc and procque are two different tags.

EXAMPLES

cfiltl -f file cpus:user:sys:nice:idle cpu+=0,1,2,3:user/4:sys/4:nice/4:idle/4

This is not useful, but demonstrates what cfiltl can do. On a 4-core system, the command extracts the user, sys, nice, and idle columns from collectl's "CPU SUMMARY" subsystem, and then for comparison, calculates the summary values using the detail CPU statistics.
Here's some output:

#timestamp cpus:user cpus:sys cpus:nice cpus:idle cpu+=0,1,2,3:user/4 cpu+=0,1,2,3:sys/4 cpu+=0,1,2,3:nice/4 cpu+=0,1,2,3:idle/4
2014:06:01:17:29:33 4 1 0 95 4.25 1.25 0.00 94.75
2014:06:01:17:29:35 2 1 0 97 3.00 1.00 0.00 96.00
2014:06:01:17:29:37 2 1 0 96 2.25 0.75 0.00 96.50
2014:06:01:17:29:39 2 1 0 97 2.75 1.00 0.00 96.00
2014:06:01:17:29:41 2 1 0 96 2.50 1.00 0.00 96.50

cfiltl -f file cpus:user:sys:intrpt#

This extracts the CPU SUMMARY fields user and sys without modification, and intrpt with normalization to 100. This allows all three statistics to be plotted on the same graph.
cfiltl -f file cpus:user+nice dsks:KbTot# nets:TxKBTot#

This allows plotting the sum of user+nice along with disk and network throughput. For example:

This was produced using gnuplot using this data and these gnuplot commands.

RESTRICTIONS

The user is expected to apply a modicum of common sense when composing tag-expressions. For example, there are some collectl columns that contain string, and not numerical values, such as the User and Command columns in the PROCESS subsystem. If you use such tags in tag-expressions, cfiltl will happily perform the requested arithmetic -- the result will always be zero.

There are still some rough edges: I haven't figured out an easy way to check whether a subsystem in an expression exists, or whether data exists in the specified file for some subsystems, and a few other things, but it seems to work.

Also, I probably need to add some exceptions for naming some subsystems in subsystem-expressions.

Because of the way collectl formats data for plotting, cfiltl currently only understands plot output containing two kinds of data:

one long line for most subsystems
one process line per process

It will probably choke if it sees environmentals data, and possibly other data for which I have no test system or example output. Candidates are: Lustre, Interconnect.

If you would like me to add support for something that is currently missing, let me know.

DOWNLOAD

Current Version

cfiltl v1.4

Previous Versions

cfiltl v1.3

cfiltl v1.2

urban[at]tru64[dot]org