Nagios: check_logfiles

Altibase is often called as hybrid database, which means store data both tablespace in a file system and memory. Normally, traditional database makes tablespace for storing data. But Altibase uses tablespace in memory. 
Memory is faster than Disk, this why we often store data into cache. 

Altimon is a monitoring deamon for this DB. Altimon periodically executes system queries which are written in configuration file. If Altimon detects one of results is over threshold, It writes error in another file. This post is for how I interact this log file with Nagios. 

1. What is check_logfiles ? 

check_logfiles is used to scan the lines of a file for regular expressions.

The plugin check_logfiles was designed to operate in mission critical environments where missing log lines after a logfile rotation could not be tolerated. 

When such a logfile rotation takes place, check_logfiles detects this and analyses the lines of the archived logfile, even if it's compressed.

It normally scans only the lines of a logfile which were added since the last run of the plugin.

The main features are:

- multiple regular expressions can be given

- expressions can be categorized as warning or critical

- it can handle any logfile rotation strategy

- hook scripts (either external scripts or a piece of perl-code in the configuration file) are possible, taking actions when a line matches a pattern. (for example, whenever a critical pattern is found, a nsca message is sent to the nagios server) 

2. Install check_logfiles & Test 

It is not able to install check_logfiles plugin via yum on CentOS. It hasn't been included its repository. It needs to download source code and install it manually. 

# Downloads check_logfiles

$ cd /downloads/

$ wget http://labs.consol.de/download/shinken-nagios-plugins/check_logfiles-3.5.1.tar.gz

$ ls -al

total 102490364

-rw-r--r--   1 root   root         138465 Dec 28  2007 check_logfiles-3.5.1.tar.gz

...

$ tar xzvf check_logfiles-3.5.1.tar.gz
$ cd check_logfiles-3.5.1

# The default directory of my nagios is  /usr/lib64/nagios/plugins/ 
$ ./configure --prefix=/usr/lib64/nagios/plugins/
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether make sets $(MAKE)... (cached) yes
variable with_seekfiles_dir is /var/tmp/check_logfiles
checking for sh... /bin/sh
checking for perl... /usr/bin/perl
checking for gzip... /bin/gzip
checking for gawk... /bin/gawk
checking for echo... /bin/echo
checking for sed... /bin/sed
checking for cat... /bin/cat
configure: creating ./config.status
config.status: creating Makefile
config.status: creating plugins-scripts/Makefile
config.status: creating plugins-scripts/subst
config.status: creating t/Makefile
                       --with-perl: /usr/bin/perl
                       --with-gzip: /bin/gzip
              --with-seekfiles-dir: /var/tmp/check_logfiles
              --with-protocols-dir: /tmp
               --with-trusted-path: /bin:/sbin:/usr/bin:/usr/sbin
                --with-nagios-user: nagios
               --with-nagios-group: nagios
                           
$ make
Making all in plugins-scripts
make[1]: Entering directory `/downloads/check_logfiles-3.5.1/plugins-scripts'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/downloads/check_logfiles-3.5.1/plugins-scripts'
Making all in t
make[1]: Entering directory `/downloads/check_logfiles-3.5.1/t'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/downloads/check_logfiles-3.5.1/t'
make[1]: Entering directory `/downloads/check_logfiles-3.5.1'
make[1]: Nothing to be done for `all-am'.
make[1]: Leaving directory `/downloads/check_logfiles-3.5.1'

$ make install
Making install in plugins-scripts
make[1]: Entering directory `/downloads/check_logfiles-3.5.1/plugins-scripts'
make[2]: Entering directory `/downloads/check_logfiles-3.5.1/plugins-scripts'
test -z "/usr/lib64/nagios/plugins/libexec" || mkdir -p -- "/usr/lib64/nagios/plugins/libexec"
/usr/bin/install -c 'check_logfiles' '/usr/lib64/nagios/plugins/libexec/check_logfiles'
make[2]: Nothing to be done for `install-data-am'.
make[2]: Leaving directory `/downloads/check_logfiles-3.5.1/plugins-scripts'
make[1]: Leaving directory `/downloads/check_logfiles-3.5.1/plugins-scripts'
Making install in t
make[1]: Entering directory `/downloads/check_logfiles-3.5.1/t'
make[2]: Entering directory `/downloads/check_logfiles-3.5.1/t'
make[2]: Nothing to be done for `install-exec-am'.
make[2]: Nothing to be done for `install-data-am'.
make[2]: Leaving directory `/downloads/check_logfiles-3.5.1/t'
make[1]: Leaving directory `/downloads/check_logfiles-3.5.1/t'
make[1]: Entering directory `/downloads/check_logfiles-3.5.1'
make[2]: Entering directory `/downloads/check_logfiles-3.5.1'
make[2]: Nothing to be done for `install-exec-am'.
make[2]: Nothing to be done for `install-data-am'.
make[2]: Leaving directory `/downloads/check_logfiles-3.5.1'
make[1]: Leaving directory `/downloads/check_logfiles-3.5.1'

$ ls -al /usr/lib64/nagios/plugins/libexec/
total 192
-rwxr-xr-x 1 root root 194274 Sep 20 15:26 check_logfiles

# Test check_files, I created file called test.log in /downloads and I added lines of words that 

# contained "ARLRM". 

$ /usr/lib64/nagios/plugins/libexec/check_logfiles --tag=altibase --logfile=/downloads/test.log --criticalpattern="ALARM"
CRITICAL - (4 errors in check_logfiles.protocol-2012-09-20-15-31-02) - [2012/06/12 00:00:12] [ALARM]:: [SESSION_COUNT.SID_COUNT] current [129] > checkValue [1] ...|altibase_lines=4 altibase_warnings=0 altibase_criticals=4 altibase_unknowns=0

# Next, added parameter "report=long", I wanted to see whole lines of errors.

/usr/lib64/nagios/plugins/libexec/check_logfiles --tag=altibase --logfile=/downloads/test.log --criticalpattern="ALARM" --report=long
CRITICAL - (6 errors in check_logfiles.protocol-2012-09-20-15-52-55) - [2012/06/12 11:38:22] [ALARM]:: [MEM_DATABASE_USE.ALLOC_MEM_MB] current [11296.09] > checkValue [7000] ...|altibase_lines=6 altibase_warnings=0 altibase_criticals=6 altibase_unknowns=0
tag altibase CRITICAL

[2012/06/12 00:00:11] [ALARM]:: [PROCESS.MEM_USAGE(KB)] Current (15456872) >= Limit (10240000)

[2012/06/12 00:00:12] [ALARM]:: [MEMSTAT_SUM.MAX_TOTAL_MB] current [17818.98] > checkValue [10240]

[2012/06/12 00:00:12] [ALARM]:: [MEM_DATABASE_USE.ALLOC_MEM_MB] current [11296.09] > checkValue [7000]

[2012/06/12 00:00:12] [ALARM]:: [SESSION_COUNT.SID_COUNT] current [129] > checkValue [1]
[2012/06/12 11:38:21] [ALARM]:: [MEMSTAT_SUM.MAX_TOTAL_MB] current [17866.01] > checkValue [10240]

[2012/06/12 11:38:22] [ALARM]:: [MEM_DATABASE_USE.ALLOC_MEM_MB] current [11296.09] > checkValue [7000]

Default parameters are follows: 

--tag= This is for Idenfication, If you want check one more combination of logfile/pattern, you should use this. (Optional) 

--logfile= log file for scan 

--criticalpattern= Regular expression for Critical 

--warningpattern= Regular expression for Warning  (Optional) 

--noprotocol=Switch off of logging match results in a separated file (default protocol files are created in /tmp

** Protocol – The matching lines can be written to a protocol file the name of which will be included in the plugin’s output. 

The path of protocol file definition can be defined in configuration file. The definitions in this file are written with Perl-syntax.

$protocolsdir: The default is /tmp or the directory which has been specified with the –with-protocol-dir of ./configure.

$protocolretention: The lifetime of protocol files in days. After these days the files are deleted automatically (default 7 days)

References: 
1. http://labs.consol.de/lang/en/nagios/check_logfiles/
2. http://exchange.nagios.org/directory/Plugins/Operating-Systems/Linux/check_logfiles/details