How to solve message "Starting nagios:No directory, logging in with HOME=/" when starting nagios deamon

I wrote a script of installing nagios 3 named "install-nagios3.sh" on my github repository (https://github.com/yeonki-choi/nagios) and tested it. After ran this, everything was fine excepts for message "Starting nagios:No directory, logging in with HOME=/" when starting nagios daemon.

$ sudo service nagios start
Starting nagios:No directory, logging in with HOME=/
done.

I found this message was same-produced when I switched to user "nagios". It was caused there was no home directory for user "nagios". This script created the user "nagios" with no home directory during running to own nagios-core's home directory and start its daemon.

$ sudo su - nagios
No directory, logging in with HOME=/

To do solve this, it just make home directory for that user

# Make a directory and changed the ownership
$ sudo mkdir /home/nagios
$ sudo chown -R nagios:nagios /home/nagios

# Setting the directory as the home of nagios
$ sudo usermod --home /home/nagios nagios
usermod: no changes

# Restarting nagios daemon, this time there is no above message 
$ sudo service nagios restart
Running configuration check...done.
Stopping nagios: done.
Starting nagios: done.

Note things in dealing with Nagios+Ndo2DB and properties in host configuration

I'd like to point out two things related nagios.
As we know, we are able to save status data of host and service into relational database using ndo2db module and I described how to setup at a post.

When people have nagios + nod2db monitoring system in their environments, they may find that data is duplicated in the table "nagios_hosts" and what makes this happen.

For example, this is result of the query of host definitions

mysql> SELECT host_id, alias, address, config_type FROM `nagios_hosts`;
+----------+-----------+----------------+---------------+
| host_id  | alias     |  address       |  config_type  |
+----------+-----------+----------------+---------------+
|10963     |test03     | 192.168.20.183 |            1  |                                 
|10962     |cent-test  | 192.168.20.175 |            1  |                                 
|10961     |FDCMOCDB01 | 192.168.20.172 |            1  |                                 
|10960     |test03     | 192.168.20.183 |            0  |                                 
|10959     |cent-test  | 192.168.20.175 |            0  |                                 
|10958     |FDCMOCDB01 | 192.168.20.172 |            0  |   
+----------+-----------+----------------+---------------+

Like this, every host is duplicated except for config_type.

Ndo2db can either get host definition from the original config file (It is usually located in /etc/nagios/objects/hosts.cfg) or retained config file(/var/log/nagios/status.dat)

As default, config_output_options is 3 which means get both data from config file and retained config file.

$ cat /etc/nagios/ndomod.cfg | grep config*
...
# CONFIG OUTPUT OPTION
# This option determines what types of configuration data the NDO NEB module will dump from Nagios.
# Values can be OR'ed together.
# Values:
#         0 = Don't dump any configuration information
#         1 = Dump only original config (from config files)
#         2 = Dump config only after retained information has been restored
#         3 = Dump both original and retained configuration
config_output_options=3

I changed this value to 2, I'd like to get data only from the retained file.

$ cat /etc/nagios/ndomod.cfg
...
config_output_options=2

Then, Restarting Services

$ service ndo2db restart
$ service nagios restart

mysql> SELECT host_id, alias, address, config_type FROM `nagios_hosts`;
+----------+-----------+----------------+---------------+
| host_id  | alias     |  address       |  config_type  |
+----------+-----------+----------------+---------------+
|10972     |test03     | 192.168.20.183 |            1  |                                 
|10971     |cent-test  | 192.168.20.175 |            1  |                                 
|10970     |FDCMOCDB01 | 192.168.20.172 |            1  |                                 
+----------+-----------+----------------+---------------+

Next topic, I think it is more important point.
I usually edit config files in the directory "/etc/nagios/objects/" when need to change some properties, active_checks_enabled for instance.
But I managed to know that nagios doesn't pick up some properties from config files. Rather, nagios relies retention files.

It is said that the information from nagios manual

"It is important to point out that several directives in host, service, and contact definitions may not be picked up by Nagios when you change them in your configuration files. Object directives that can exhibit this behavior are marked with an asterisk (*). The reason for this behavior is due to the fact that Nagios chooses to honor values stored in the state retention file over values found in the config files, assuming you have state retention enabled on a program-wide basis and the value of the directive is changed during runtime with an external command."

And these are directives for host definition
...
active_checks_enabled *: ....
passive_checks_enabled *: ....
...

The problem is that I cannot edit the retention status file which is read-only file and only editable by nagios. So, how can I do to change unchangeable properties by ending in config file in programming manner not using nagios web.

Naiogs provides cmd.cgi interface and it can run like the below:

$ curl -d "cmd_mod=2&cmd_typ=48&host=FDCMOCDB01" "http://localhost/nagios/cgi-bin/cmd.cgi" -u "nagiosadmin:*******"
<html>
<head>
<link rel="shortcut icon" href="/nagios/images/favicon.ico" type="image/ico">
<title>
External Command Interface
</title>
<LINK REL='stylesheet' TYPE='text/css' HREF='/nagios/stylesheets/common.css'>
<LINK REL='stylesheet' TYPE='text/css' HREF='/nagios/stylesheets/cmd.css'>
</head>
<body CLASS='cmd'>

<!-- Produced by Nagios (http://www.nagios.org).  Copyright (c) 1999-2007 Ethan Galstad. -->
<table border=0 width=100%>
<tr>
<td align=left valign=top width=33%>
<TABLE CLASS='infoBox' BORDER=1 CELLSPACING=0 CELLPADDING=0>
<TR><TD CLASS='infoBox'>
<DIV CLASS='infoBoxTitle'>External Command Interface</DIV>
Last Updated: Tue Jul 9 16:04:13 KST 2013<BR>
Nagios® Core™ 3.3.1 - <A HREF='http://www.nagios.org' TARGET='_new' CLASS='homepageURL'>www.nagios.org</A><BR>
Logged in as <i>nagiosadmin</i><BR>
</TD></TR>
</TABLE>
</td>
<td align=center valign=top width=33%>
</td>
<td align=right valign=bottom width=33%>
</td>
</tr>
</table>
<P><DIV CLASS='infoMessage'>Your command request was successfully submitted to Nagios for processing.<BR><BR>
Note: It may take a while before the command is actually processed.<BR><BR>
<A HREF='javascript:window.history.go(-2)'>Done</A></DIV></P>
<!-- Produced by Nagios (http://www.nagios.org).  Copyright (c) 1999-2007 Ethan Galstad. -->
</body>
</html>

This is how to run cmd.cgi with curl in the shell environment

$ curl -d cmd_typ=${cmd} \
      -d cmd_mod=2 \
      -d host=${host} \
      -d service=${svc} \
      -d btnSubmit=Commit \
      -s \
      -u 'user:pass \
      "http://nagios.server/nagios/cgi-bin/cmd.cgi"

As parameter "cmd_typ", constant values are defined in the header file "common.h"

...
#define CMD_ENABLE_HOST_CHECK 47
#define CMD_DISABLE_HOST_CHECK 48
...

If you implement this in java, you are able to execute it by using httpclient class (package org.apache.commons.httpclient) like my case.

References:
1. http://eggsonbread.com/2011/03/18/disable-enable-nagios-notifications-via-command-line-curl/
2. https://github.com/ageric/nagios/blob/cc67733e82721546d9d04180c4441cdb41a6f9a2/include/common.h
3. http://forums.cacti.net/post-133568.html

Book review: Nagios Core Administration Cookbook

I've involved in a cloud development project. I use Nagios core for monitoring servers and devices in this project. As we know, Nagios is one of famous open sourced monitoring tools. I needed references in dealing with Nagios. While I was reading this, it remind me of basic concept of Nagios again. 



There are some sections in every chapter. "How to do it" , "How it works" and "There's more". it describes how we start a certain feature of Nagios according to the section in "How to do it" and then explains its mechanism with easy way
in "How it works" and finally there is additional contents related to the chapter in "There's more"

There is impressive paragraph in the book that nagios is close to MONITORING FRAMWORK rather than monitoring tool. It is perfect that describes nagios characteristic because it has already provide ecosystem, for example, lots of plugins support . And it is important to know where to find if need specific plugin that doesn't meet requirements. Author instructed that how to find a plugin that the administrator needs and where. 

People who are unfamiliar with Nagios, in my opinion, they may be confused that Nagios has SOFT and HARD state in response to server or host failing. In chapter 3, author tries to explain the difference between two by giving an understandable example. 

At last, in chapter 11, he gives an explanation about add-on projects like NRPE, NSClient++, NDOUtils and SNMP trap. If you need to extend the feature of Nagios to provide better-fitting solution on your own production, this will very helpful to you.

Book link: http://www.packtpub.com/nagios-core-administration-cookbook/book

Nagios NDO2DB: Unicode problem

I had encountered a problem related NDO2DB. I configured some of services' name  in Korean character. It shows well in Nagios web, but these services' name could't be recognized in Mysql tables which Ndo2db saves into becuase Ndo2db  add-on processes data without considering uni-code data. It was not solved even though I changed character set to UTF-8 in the tables.     

One of possible way to deal with is to change convert column which have to show by UTF8 in SQL query. 

The solutions is as following: 
CONVERT(CAST(column_name) as BINARY) USING UTF8) 

"This is a similar approach to our SELECT CONVERT(CAST(city as BINARY) USING utf8) trick above, where we basically hide the column’s actual data from MySQL by masking it as BINARY temporarily. 

If you simply force the column to UTF-8 without the BINARY conversion, MySQL does a data-changing conversion of your “latin1″ characters into “UTF-8″ and you end up with improperly converted data. Converting the column to BINARY first forces MySQL to not realize the data was in UTF-8 in the first place."

I referenced this from this site: http://nicj.net/mysql-converting-an-incorrect-latin1-column-to-utf8/

Nagios: How to change datetime format of snmp trap

After I installed and Configured SNMPTT, saving snmpt trap data into table worked well.
Additionally, I needed to change the date format so that I could filter them easily by date and time.

By default, date format should be "Sun Dec 16 15:23:47 2012", but I want it to look like "2012-12-16 15:23:47"

When look at /etc/snmp/snmptt.ini, there are some properties related to datetime format.

$ vi /etc/snmp/snmptt.ini
...
# The date time format in strftime() format for the date/time when logging to 
# standard output, snmtt log files (log_file) and the unknown log file
# (unknown_trap_log_file). Defaults to localtime(). For SQL, see
# date_time_format_sql.
# Example:  %a %b %e %Y %H:%M:%S
# date_time_format = 
...
...
# The date time format for the traptime column in SQL, Defaults to 
# localtime(). When a date/time field is used in SQL, this should be 
# changed to follow a standard that is supported by the SQL server.
# Example:  For a MySQL DATETIME, use %Y-%m-%d %H:%M:%S.
# date_time_format_sql = 
...

As we can know from the above description, snmptt uses strftime() format.
The first format definition (date_time_format) is for log files. Generally, /var/log/snmptt/snmptt.log and /var/log/snmptt/snmpttunknown.log. The second definition (date_time_format_sql) is for traptime column of snmptt table in MySQL.

Before change the format in snmptt.ini, you can validate the format like the following:

$ mydate=`perl -e 'use POSIX; print strftime "%Y-%m-%d %T", localtime time;'`
$ echo $mydate
2013-01-22 10:19:45

Then uncomment those lines and change the formats in snmptt.ini

$ vi /etc/snmp/snmptt.ini
...
date_time_format = %Y-%m-%d %T
...
date_time_format_sql = %Y-%m-%d %T

# It needs to restart
$ service snmptt restart

Please visit the site to see various date time formats in strftime
http://my.safaribooksonline.com/book/operating-systems-and-server-administration/bash/0596526784/reference-lists/date_and_time_string_formatting_with_strftime

References:
1. http://stackoverflow.com/questions/13093709/how-to-use-shell-variables-in-perl-command-call-in-a-bash-shell-script

Nagios SNMP Trap: part 2 - Configuration

In my last post, I described how to install Nagios core, plugin and Snmptt. This time I'm going to focus on explanation of configuration.

This step is converting CISCO MIB file (it is omitted, In my opinion, it should have showed on my last).

# ConvertMib of CISCO MIB file
$ snmpttconvertmib --in=/usr/share/snmp/mibs/CISCO-UNIFIED-COMPUTING-MIB.my --out=/etc/snmp/snmptt.conf.cisco --exec='/usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1' --net_snmp_perl
exec: /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1

*****  UCD-SNMP / NET-SNMP Perl module enabled *****
****  Processing MIB file *****

snmptranslate version: NET-SNMP version: 5.5
severity: Normal

File to load is:        /usr/share/snmp/mibs/CISCO-UNIFIED-COMPUTING-MIB.my
File to APPEND TO:      /etc/snmp/snmptt.conf.cisco

MIBS environment var:   /usr/share/snmp/mibs/CISCO-UNIFIED-COMPUTING-MIB.my
mib name: CISCO-UNIFIED-COMPUTING-MIB

*****  Using UCD-SNMP / NET-SNMP Perl module *****

Processing MIB:         CISCO-UNIFIED-COMPUTING-MIB
#
Split line TRAP-TYPE / NOTIFICATION-TYPE found (MODULE-IDENTITY,).
Line: 15
NOTIFICATION-TYPE: MODULE-IDENTITY
Enterprise: ciscoMgmt
Looking up via snmptranslate: CISCO-UNIFIED-COMPUTING-MIB::MODULE-IDENTITY
Unknown object identifier: CISCO-UNIFIED-COMPUTING-MIB::MODULE-IDENTITY
OID:
#
Line: 2216
NOTIFICATION-TYPE: cucsFaultActiveNotif
Variables: cucsFaultDescription cucsFaultAffectedObjectId cucsFaultAffectedObjectDn cucsFaultCreationTime cucsFaultLastModificationTime cucsFaultCode cucsFaultType cucsFaultProbableCause cucsFaultSeverity cucsFaultOccur
Enterprise: ciscoUnifiedComputingMIBNotifs
Looking up via snmptranslate: CISCO-UNIFIED-COMPUTING-MIB::cucsFaultActiveNotif
OID: .1.3.6.1.4.1.9.9.719.0.1
#
Line: 2235
NOTIFICATION-TYPE: cucsFaultClearNotif
Variables: cucsFaultDescription cucsFaultAffectedObjectId cucsFaultAffectedObjectDn cucsFaultCreationTime cucsFaultLastModificationTime cucsFaultCode cucsFaultType cucsFaultProbableCause cucsFaultSeverity cucsFaultOccur
Enterprise: ciscoUnifiedComputingMIBNotifs
Looking up via snmptranslate: CISCO-UNIFIED-COMPUTING-MIB::cucsFaultClearNotif
OID: .1.3.6.1.4.1.9.9.719.0.2

Done
Total translations:        3
Successful translations:   2
Failed translations:       1

/etc/snmp/snmptt.conf.cisco file is successfully created.

Next, it is configuration of Nagios.

# Create Nagios Service for SNMP TRAP
$ vi /usr/local/nagios/etc/objects/remote-host.cfg

# Define a host for the remote machine
define host {
        use     linux-server
        host_name       cisco-ucs
        alias           cisco ucs manager
        address         192.168.20.228
        }

# Define a service to "trap-service" the server
define service{
        use                        generic-service
        name                       trap-service
        register                   0
        service_description        TRAP
        is_volatile                1
        check_command              check_host_alive
        max_check_attempts         1
        normal_check_interval      1
        retry_check_interval       1
        notification_interval      31536000
        active_checks_enabled      0
        passive_checks_enabled     1
        }

# Define a service to check
define service{
        use                        trap-service
        host_name                  cisco-ucs
        }

$ service nagios restart

MySQL Server installation, I assume that I already installed MySQL. so I just created SMNPTT tables on nagios_db (Originally, nagios_db is used store ndo2db data and I simply add tables without creating DB). Table scripts are copied from here.

# Crate new user "snmptt"
grant all privileges on nagios_db.* to 'snmptt'@'localhost' identified by 'snmptt';
mysql> grant all privileges on nagios_db.* to 'snmptt'@'%' identified by 'snmptt';

# Add tables into the database
CREATE TABLE snmptt (
id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
eventname VARCHAR(50),
eventid VARCHAR(50),
trapoid VARCHAR(100),
enterprise VARCHAR(100),
community VARCHAR(20),
hostname VARCHAR(100),
agentip  VARCHAR(16),
category VARCHAR(20),
severity VARCHAR(20),
uptime  VARCHAR(20),
traptime VARCHAR(30),
formatline VARCHAR(255));

CREATE TABLE snmptt_unknown (
trapoid VARCHAR(100),
enterprise VARCHAR(100),
community VARCHAR(20),
hostname VARCHAR(100),
agentip  VARCHAR(16),
uptime  VARCHAR(20),
traptime VARCHAR(30),
formatline VARCHAR(255));

CREATE TABLE snmptt_statistics (
stat_time VARCHAR(30),
total_received BIGINT,
total_translated BIGINT,
total_ignored BIGINT,
total_unknown BIGINT);

# Needs to install additional perl modules
# Download ftp://ftp.perl.org/pub/CPAN/authors/id/T/TI/TIMB/DBI-1.622.tar.gz and ftp://ftp.perl.org/pub/CPAN/authors/id/T/TI/TIMB/CHECKSUMS
# Copy to /root/.cpan/sources/authors/id/authors/id/T/TI/TIMB/
$ perl -MCPAN -e 'install DBI'

# Download ftp://ftp.perl.org/pub/CPAN/authors/id/C/CA/CAPTTOFU/DBD-mysql-4.022.tar.gz and ftp://ftp.perl.org/pub/CPAN/authors/id/C/CA/CAPTTOFU/CHECKSUMS
# Copy to  /root/.cpan/sources/authors/id/C/CA/CAPTTOFU/
$ perl -MCPAN -e 'install DBD::mysql'

$ vi /etc/snmp/snmptt.ini
mysql_dbi_enable=1
mysql_dbi_host=localhost
mysql_dbi_port=3306
mysql_dbi_database=nagios_db
mysql_dbi_table=snmptt
mysql_dbi_table_unknown=snmptt_unknown
mysql_dbi_username=snmptt
mysql_dbi_password=snmptt

Those are two important tables here, snmptt table only stores data coming from H/Ws which have already registered MIB translating (In the example, CISCO UCS is my registered H/W)
On the other hand, snmptt_unknown table stores other H/Ws which are not registered MIB files (If you have Oracle Tape Backup Library and you get event trap from this H/W, the data will be stored into snmptt_unknown.

$ service snmptt restart

$ mysql -u snmptt -p nagios_db;
mysql> select id, eventname, severity from snmptt;
id   |     eventname                 | severity
1    |     EventMonitorTrapInfo  | Normal

Finally, we are able to see an eventrap data in mysql.

Troubleshooting
Problem: If you have an error, while trap data in inserted into the table:
Error : mysql_error: 'Column 'next_notification' cannot be null'

Resolution : Change the schema of the column of the table.

mysql> alter table nagios_servicestatus modify column next_notification datetime default '0000-00-00 00:00:00';
mysql> desc nagios_servicestatus;
----------------------------------------------------
Field                  |    Type     | Null | Key | Default                      | Extra
----------------------------------------------------
.....
next_notification |   datetime | YES |       | 0000-00-00 00:00:00 | 
.....

References:
1. http://forum.centreon.com/archive/index.php/t-6766.html

Nagios SNMP Trap: part 1 - SNMPTT installation

This post is going to be separated into two parts. The first part is describing to install Nagios core, plugins and SNMPTT(SNMP Trap Translator), and the second part is going to be posted about registering SNMP evethandler to make Nagios aware these messages, converting MIB files and integrating MySQL database to store messages.

SNMPTT is module to handle snmp trap message and written in perl. The follow describes how it works:

[Figure] SNMPTT Flows (http://snmptt.sourceforge.net/about.shtml)

Install Nagios core

# pre-requisite 
$ yum install -y wget httpd php gcc glibc glibc-common gd gd-devel make net-snmp*

# main module download 
$ mkdir /downloads
$ cd /downloads
$ wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.4.1.tar.gz
$ tar -xzvf ./nagios-3.4.1.tar.gz

# Adding the Nagios User and Group
$ useradd nagios
$ groupadd nagios
$ usermod -a -G nagcmd nagios

$ cd ./nagios
$ ./configure --with-command-group=nagios
*** Configuration summary for nagios 3.4.1 05-11-2012 ***:
....
General Options:
-------------------------
        Nagios executable:  nagios
        Nagios user/group:  nagios,nagios
       Command user/group:  nagios,nagios
            Embedded Perl:  no
             Event Broker:  yes
        Install ${prefix}:  /usr/local/nagios
                Lock file:  ${prefix}/var/nagios.lock
   Check result directory:  ${prefix}/var/spool/checkresults
           Init directory:  /etc/rc.d/init.d
  Apache conf.d directory:  /etc/httpd/conf.d
             Mail program:  /bin/mail
                  Host OS:  linux-gnu

Web Interface Options:
------------------------
                 HTML URL:  http://localhost/nagios/
                  CGI URL:  http://localhost/nagios/cgi-bin/
Traceroute (used by WAP):

Review the options above for accuracy.  If they look okay,
type 'make all' to compile the main program and CGIs.

# Install main program
$ make all
$ make install 

# Makes auto start script
$ make install-init  

# Make config files 
$ make install-config

# install external command file
$ make install-commandmode

# Configure nagios web and connect to apache 
$ make install-webconf

# Create an user of WebUI
$ htpasswd -c /usr/local/nagios/etc/htpasswd.users nagiosadmin
New password:
Re-type new password:
Adding password for user nagiosadmin

Install Nagios plugins

# Download plugins and install.
$ wget http://prdownloads.sourceforge.net/sourceforge/nagiosplug/nagios-plugins-1.4.16.tar.gz
$ tar -xzvf nagios-plugins-1.4.16.tar.gz
$ cd nagios-plugins-1.4.16
$ ./configure --with-nagios-user=nagios --with-nagios-group=nagios
$ make
$ make install
[....]
check_http.c:312: error: ‘ssl_version’ undeclared (first use in this function)
check_http.c:312: error: (Each undeclared identifier is reported only once
check_http.c:312: error: for each function it appears in.)
make[1]: *** [check_http.o] Error 1
make[1]: Leaving directory `/downloads/nagios-plugins-1.4.16/plugins'
make: *** [install-recursive] Error 1

# Need to install openssl module
$ yum install openssl-devel
$ make clean
$ ./configure --with-nagios-user=nagios --with-nagios-group=nagios
$ make
$ make install

# Registering these as AutoStart
$ chkconfig nagios on
$ chkconfig httpd on

SNMPTT (SNMP Trap Translator) installation

# download  snmptt_1.3.tgz from http://sourceforge.net/projects/snmptt/
$ snmptt_1.3.tgz
$ tar -xzvf ./snmptt_1.3.tgz
$ cd snmptt_1.3

# Copy snmptt to /usr/sbin/ and ensure it is executable (chmod +x snmptt)
$ cp ./snmptt /usr/sbin/
$ chmod a+x /usr/sbin/snmptt

# Copy snmptthandler to /usr/sbin/ and ensure it is executable (chmod +x snmptthandler)
$ cp ./snmptthandler /usr/sbin/
$ chmod a+x /usr/sbin/snmptthandler 

# Copy snmpttconvert to /usr/sbin/ and ensure it is executable (chmod +x snmptthandler)
$ cp ./snmpttconvert /usr/sbin/
$ chmod a+x /usr/sbin/snmpttconvert 

# Copy snmpttconvertmib to /usr/sbin/ and ensure it is executable (chmod +x snmptthandler)
$ cp ./snmpttconvertmib /usr/sbin/
$ chmod a+x /usr/sbin/snmpttconvertmib

# Copy snmptthandler to /usr/sbin/ and ensure it is executable (chmod +x snmptthandler)
$ cp ./snmptthandler /usr/sbin/
$ chmod a+x /usr/sbin/snmptthandler 

# Copy snmptt.ini to /etc/snmp/ or /etc/ and edit the options inside the file.
$ cp ./snmptt.ini /etc/snmp/

# Copy snmptt.ini to /etc/snmp/ or /etc/ and edit the options inside the file.
$ mkdir /var/log/snmptt

# Modify the Net-SNMP snmptrapd.conf file by adding the following line:
$ vi /etc/snmp/snmptrapd.conf
traphandle default /usr/sbin/snmptthandler

# Create the spool folder
$ mkdir /var/spool/snmptt/

# Copy the script to the init.d directory
$ cp snmptt.init.d /etc/rc.d/init.d/snmptt

Here, SNMPTT uses some perl modules (CPAN: Comprehensive Perl Archive Network - http://www.cpan.org/misc/cpan-faq.html ) and perl CPAN not installed by default on CentOS 6.

Notice: Perl modules may be installed via YUM, but some of them are not included in YUM or needed higher version number than dependencies files in YUM

$ yum install net-snmp-perl, perl-CPAN

This step is to install Config::InitFiles and List::MoreUtils, I downloaded needed files in advance, because I cannot connected to internet.
# (1) Download ftp://ftp.perl.org/pub/CPAN/authors/01mailrc.txt.gz and copy to 
/root/.cpan/sources/authors/
# (2) Download ftp://ftp.perl.org/pub/CPAN/modules/02packages.details.txt.gz and copy to 
/root/.cpan/sources/modules/
# (3) Download ftp://ftp.perl.org/pub/CPAN/modules/03modlist.data.gz and copy to 
/root/.cpan/sources/modules/
# (4) Download ftp://ftp.perl.org/pub/CPAN/authors/id/A/AD/ADAMK/List-MoreUtils-0.33.tar.gz, ftp://ftp.perl.org/pub/CPAN/authors/id/A/AD/ADAMK/CHECKSUMS and copy to 
/root/.cpan/sources/authors/id/A/AD/ADAMK/
# (5) Download ftp://ftp.perl.org/pub/CPAN/authors/id/S/SH/SHLOMIF/Config-IniFiles-2.78.tar.gz, ftp://ftp.perl.org/pub/CPAN/authors/id/S/SH/SHLOMIF/CHECKSUMS and copy to 
/root/.cpan/sources/authors/id/S/SH/SHLOMIF/
# (6) Download ftp://ftp.perl.org.pub.CPAN/authors/id/M/MS/MSCHWERN/CHECKSUMS/Test-Simple-0.98.tar.gz, ftp://ftp.perl.org.pub.CPAN/authors/id/M/MS/MSCHWERN/CHECKSUMS
/root/.cpan/sources/authors/id/M/MS/MSCHWERN/

$ perl -MCPAN -e 'install List::MoreUtils'

$ perl -MCPAN -e 'install Config::InitFiles'

# If you have en error "Can't locate Module/Build/Compat.pm in @INC contain...", you should install the following module to solve dependencies.  
$ yum install perl-Moudle-Build

Put it simply, the connection between Nagios and snmptt is as follows:


As can see how it works from the above image. When the server triggers trap messages, snmptrapd is the first to handle them on monitoring server. Look at the following script. It defines that handling of trap is delegating to snmptt in the configuration.

$ vi /etc/snmp/snmptrapd.conf
...
traphandle default /usr/sbin/snmptt
...


$ vi /etc/rc.d/init.d/snmptrapd
....
OPTIONS="-On -Lsd -p /var/run/snmptrapd.pid"
....

Next, snmptt makes decision whether the trap message is known or unknown based on predefined trapfiles:

$ vi /etc/snmp/snmptt.ini
...
[TRAPFILES]
snmptt_conf_files = <<END
/etc/snmp/snmptt.conf.cisco
/etc/snmp/snmptt.conf.emc
END
...

The first part is finished at here. I originally decided not to divide, but I realized this would be long one for one blog. I'll post the second part of this as soon as possible.

Here's the link about part 2: http://hyper-choi.blogspot.kr/2013/01/nagios-snmp-trap-part-2-configuration.html


References:
1. Installation manual: http://assets.nagios.com/downloads/nagioscore/docs/Installing_Nagios_Core_From_Source.pdf
2. http://www.thesysadminhimself.com/2012/09/compiling-nagios-plugins-1416-throws.html
3. http://www.linuxquestions.org/questions/linux-newbie-8/nagios-internal-server-error-903798/
4. http://www.linuxquestions.org/questions/linux-software-2/snmp-monitoring-with-nagios-645270/
5. http://it.bogema.nl/tag/centos-6-2/
6. http://askaralikhan.blogspot.kr/2010/12/receiving-snmp-traps-in-nagios.html

Nagois Eventhandler

I wrote about how to resister remote process in nagios monitoring environment in my last writing. This time, I'd like to post about registering a action. The action is that when a process is sopped or is killed unexpectedly, then we should trgger the action which is sort of restart the killed process. In nagios, it is able to make action, which is called 'eventhandler'.

As I was previously mentioned, my VMs were continusly shutting down and then I made services for monitoring if they were dead or alive. Now, it is time to make a eventhandler for actions. There are some important points to remember in configuring the eventhandler.

   - The property 'max_checks_attempts' in nagios.config
   - StateType: SOFT STATE vs HARD STATE

Nagios comes to conclusion times when it calls the eventhandler or gives the administrator notification by these points. When monitoring a service and its state has been changed into 'Critical', nagios only calls eventhandler until the check time is reached to ckeck max_checks_attempts (the service is in SOFT state). If total check time is reached to max_check_attemps, nagios notifies the failure of the service (service is now in HARD state).

The following link is descriptive well about statetype: http://nagios.sourceforge.net/docs/3_0/statetypes.html

Resistering eventhandler

1) Wiring a script

$ vi /usr/lib64/nagios/plugins/eventhandlers/restart-vm

#!/bin/sh
#
# Event handler script for restarting the nrpe server on the local machine
#
# Note: This script will only restart the nrpe server if the service is
#       retried 3 times (in a "soft" state) or if the web service somehow
#       manages to fall into a "hard" error state.
#

date=`date`

case "$1" in
OK)
 # The service just came back up, so don't do anything...
 ;;
WARNING)
 # We don't really care about warning states, since the service is probably still running...
 ;;
UNKNOWN)
 # We don't know what might be causing an unknown error, so don't do anything...
 ;;
CRITICAL)
 # Aha!  The BLAH service appears to have a problem - perhaps we should restart the server...

 # Is this a "soft" or a "hard" state?
 case "$2" in

 # We're in a "soft" state, meaning that Nagios is in the middle of retrying the
 # check before it turns into a "hard" state and contacts get notified...
 SOFT)

  # What check attempt are we on?  We don't want to restart the web server on the first
  # check, because it may just be a fluke!
  case "$3" in

  # Wait until the check has been tried 2 times before restarting the web server.
  # If the check fails on the 3rd time (after we restart the web server), the state
  # type will turn to "hard" and contacts will be notified of the problem.
  # Hopefully this will restart the web server successfully, so the 4th check will
  # result in a "soft" recovery.  If that happens no one gets notified because we
  # fixed the problem!
  2)
   echo -n "Restarting the VM service (3rd soft critical state)..."
   # Call the init script to restart the VM
   /usr/bin/sudo /usr/sbin/xm start $4
   echo "$date - restart $4 - SOFT"  >> /tmp/eventhandlers
   ;;
   esac
  ;;

 # The BLAH service somehow managed to turn into a hard error without getting fixed.
 # It should have been restarted by the code above, but for some reason it didn't.
 # Let's give it one last try, shall we?
 # Note: Contacts have already been notified of a problem with the service at this
 # point (unless you disabled notifications for this service)
 HARD)
  case "$3" in

  4)
   echo -n "Restarting VM  service..."
   # Call the init script to restart the VM
   echo "$date - restart $4 - HARD"  >> /tmp/eventhandlers
   /usr/bin/sudo /usr/sbin/xm start $4
   ;;
   esac
  ;;
 esac
 ;;
esac
exit 0

I was writing the script file that made nagios call this the while my VM was in 'critical'. I set max_check_attempts to 3 in config file and defined to start VM when it became the second and the third continuous fail.

2) The following works that giving execution permission to the script and add a command, service to nagios:

# give execution permission to the script
$ chmod a+x /usr/lib64/nagios/plugins/eventhandlers/restart-vm

# define the eventhandler as new command
$ vi /etc/nagios/objects/commands.cfg
define command{
command_name    restart-vm
command_line    /usr/lib64/nagios/plugins/eventhandlers/restart-vm $SERVICESTATE$ $SERVICESTATETYPE$ $SERVICEATTEMPT$ $SERVICEDESC$
}

# set the command to a specific service
$ vi /etc/nagios/objects/services.cfg
define service{
use generic-service
host_name localhost
hostgroup_name
service_description ad
active_checks_enabled 1
passive_checks_enabled 1
check_command check_local_procs!1:1!qemu-dm!'domain-name ad'
event_handler restart-vm
}

# Nagios always needs to be validation and restart after modified config files.
$nagios -v /etc/nagios/nagos.cfg
$service nagios restart

When I looked at nagios.debug file (/var/log/nagios/nagios.debug), they were said like:

[1354857538.204205] [256.1] [pid=6862] Running command '/usr/lib64/nagios/plugins/eventhandlers/restart_vm CRITICAL SOFT 1 ad'...
[1354857538.251733] [256.1] [pid=6862] Execution time=0.047 sec, early timeout=0, result=0, output=(null)

[1354857548.035316] [256.1] [pid=6862] Running command '/usr/lib64/nagios/plugins/eventhandlers/restart_vm CRITICAL SOFT 2 ad'...

[1354857550.770515] [256.1] [pid=6862] Execution time=2.734 sec, early timeout=0, result=0, output=Restarting the VM service (2nd soft critical state)...Fri Dec  7 14:19:08 KST 2012 - restart ad - SOFT
[1354857558.040135] [256.1] [pid=6862] Running command '/usr/lib64/nagios/plugins/eventhandlers/restart_vm OK SOFT 3 ad'...

Troubleshooting
If the script doesn't work, check where the execution is given to the script file.

$ ls -al /usr/lib64/nagios/plugins/eventhandlers/restart-vm
-rwxr-xr-x 1 root root 3032 Dec  7 14:05 /usr/lib64/nagios/plugins/eventhandlers/restart-vm

Next is checking /etc/sudoers. In my case, the user "nagios" is the default user of Nagios monitoring server and give the permission of executing sudo without password and Defaults:nagios !requiretty

$ vi /etc/sudoers
Defaults:nagios !requiretty
nagios          ALL=(ALL)       NOPASSWD: ALL

References:
1. http://www.techadre.com/content/nagios-event-handler-restarting-local-service
2. http://forums.meulie.net/viewtopic.php?f=59&t=5918

Posted in Blogsy (iPad)

Nagios: Monitor processes in remote host using regular expression

I had encountered the problem that my VMs became repeatedly shutdown. Until now, I couldn't the solve this, but I had to work around of it. I took the second best for this was to detect the downtime of VMs and then started them by Nagios.

check_procs plugin is used to monitor specific process count. It is able to set command name with -C parameter to watch the status of the process and when also able to set arguments with -a.

check_procs can be executed the following: 

$ /usr/lib64/nagios/plugins/check_procs -c 1:1 -C altibase -a 'boot from admin'
PROCS OK: 1 process with command name 'altibase', args 'boot from admin'

-c parameter is to check the ranged of process count. Here, I set to raise critical error if the count wasn't within from 1 to 1, -C parameter is for command name and -a parameter for arguments of the command, this is useful to distinguish one when some processes were found with the same command name. 

Command usage: 

check_procs -w -c [-m metric] [-s state] [-p ppid] [-u user] [-r rss] [-z vsz] [-P %cpu] [-a argument-array] [-C command] [-t timeout] [-v]

However, check_procs only monitor processes in local server. To do monitor processes in remote manner, it needs to send check_procs command to the remote host through check_nrpe.

1) Define the new command name "check_process" in a monitored server.

$ vi /etc/nagios/nrpe.cfg
command[check_process]=/usr/lib64/nagios/plugins/check_procs $ARG1$

2) Define the new command in Nagios monitoring server

$ vi /etc/nagios/object/commands.cfg
# 'check_process' command definition
define command{
  command_name    check_process
  command_line    $USER1$/check_nrpe -H $HOSTADDRESS$ -c check_process -a $ARG1$
  }

It was simple definition, I wanted to execute check_process with the argument, which is  passed though $ARG1& variable.  

Then I executed the command this on the monitoring server.

$ /usr/lib64/nagios/plugins/check_nrpe -H 192.168.20.174 -c check_process -a '-c 1:1 -C altibase -a 'boot from admin'' 
PROCS OK: 1 process with command name 'altibase', args 'boot' 

3) It worked, but the result was different
It resulted well, However, it was somewhat different with my expectation. It only with the argument 'boot', I needed to do it with the all arguments. 

So, I tried many times to apply all args but they were not succeeded at all. It  made my mind to find other ways. Indeed, check_procs supports finding process with regular expression, using --ereg-argument-array. 

I changed the command and it was succeeded: 

$ /usr/lib64/nagios/plugins/check_nrpe -H 192.168.20.174 -c check_process -a '-c 1:1 -C altibase --ereg-argument-array='boot.from.admin0''
PROCS OK: 1 process with command name 'altibase', regex args 'boot.from.admin'

In regular expression, A dot(.) means it represents any one word, for example, 'l..e' can be 'love' or 'life' etc. 

References: 

1. http://linuxsysadminblog.com/2009/02/nagios-how-to-check-if-remote-process-is-running/

2. http://blog.naver.com/choibit?Redirect=Log&logNo=140036351943

Posted in Blogsy (iPad)

Get mac address of the remote host

How can I do to get mac address of the remote server?

This question came from implementation of functionality that registering a new host into the management solution. The management server needs mac address of every new server to issue UID. 

I thought connecting to the remote host by ssh and run some commands. 
However, Windows server is including into environment. ssh doesn't work in Windows OS.   

I got an idea of using ARP(Address Resolution Protocol) from: http://windowstipoftheday.blogspot.kr/2006/02/finding-mac-address-of-remote-computer.html

1) Ping checking to find mac address (Here, I used check_ping command, one of nagios plugins)
$ /usr/lib64/nagios/plugins/check_ping -H 192.168.20.169 -w 3000.0,80% -c 5000.0,100% -p 5
PING OK - Packet loss = 0%, RTA = 1.57 ms|rta=1.574000ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0

2) If ping was succeed, run "arp -a ip" 
$ arp -a 192.168.20.169 | egrep -o '([0-9a-f]{2}:){5}[0-9a-f]{2}'
00:16:3e:c3:7a:43

References: 
1. http://windowstipoftheday.blogspot.kr/2006/02/finding-mac-address-of-remote-computer.html
2. http://www.grymoire.com/Unix/Regular.html

Nagios: NRPE - Unable to read output

I added a script that checks CPU usage for remote Linux machines. I downloaded the file and give execution permission to it.

# Download the script for CPU usage 
http://verypowerful.info/home/nagiosplugin-checkcpuperformance

# Giving a permission 
$ chmod a+x /usr/lib64/nagios/plugins/check_cpu_perf.sh

# Command definition in nrpe.cfg on the remote server.
$ vi /etc/nagios/nrpe.cfg 
command[check_cpu]=/usr/lib64/nagios/plugins/check_cpu_perf.sh $ARG1$ $ARG2$


While I was testing this script on nagios monitoring server, I got the following error: 
$ /usr/lib64/nagios/plugins/check_nrpe -H 192.168.20.174 -c check_cpu -a 30 20
NRPE: Unable to read output

At the start, I searched why this happened and how this solved. 
Most of suggestions on those results were: 
1) Add the command that you want execute into nrpe.cfg 
command[check_cpu]=sudo /usr/lib64/nagios/plugins/check_cpu_perf.sh
2) Add nagios user and the script into /etc/sudoers
nagios ALL=(ALL) NOPASSWD:/usr/lib64/nagios/plugins/check_cpu_perf.sh
3) Comment out the section in /etc/sudoers
#Defaults requiretty

However, my case was different to these. It caused because of SELINUX.
SELINUX was enabled on my remote server.


$ vi /etc/selinux/config 
....
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
# SELINUX=enforcing 
SELINUX=disabled 

# Needs to reboot to apply this.
$ reboot

I executed check_cpu again on Nagios Monitoring host
$ /usr/lib64/nagios/plugins/check_nrpe -H 192.168.20.174 -c check_cpu -a 30 20
OK: CPU Idle = 99.79% | CpuUser=0.12; CpuNice=0.00; CpuSystem=0.08; CpuIowait=0.01; CpuSteal=0.00; CpuIdle=99.79:20:10

It worked. If you can't find any solution about the error "Unable to read output", It's good to check SELINUX.  


In addition, when you have the following error message, It needs to enable "dont_blame_nrpe " option.
CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.

vi /etc/nagios/nrpe.cfg
...
dont_blame_nrpe=1

References: 
1. http://stackoverflow.com/questions/10542648/python-unable-to-call-sudo-using-popen

Nagios: Adding additional host

Nagios has configuration files that are written to manage servers. Generally, they are divided into two files: One is for Linux server and the other is for Windows. 

Server group can be defined to manage one more servers that have similar purposes. Looking at the configuration file for the sever. 

For a start, I defined a windows server. At that time, I didn't use any hostgroup.

$ vi /etc/nagios/objects/windows.cfg

define host{
        use             windows-server  ;  Inherit default values from a template
        host_name       winserver       ; The name we're giving to this host
        alias           My Windows Server  ; A longer name 
        address         192.168.20.169  ;   IP address of the host
        }

###################################################
###################################################
#
# HOST GROUP DEFINITIONS
#
###################################################
###################################################

# Define a hostgroup for Windows machines
# All hosts that use the windows-server template will automatically be a member of this group

define hostgroup{
        hostgroup_name  windows-servers ; The name of the hostgroup
        alias           Windows Servers ; Long name of the group
        }

###################################################
###################################################
#
# SERVICE DEFINITIONS 
#
###################################################
###################################################

# Create a service for monitoring the version of NSCLient++ that is installed
# Change the host_name to match the name of the host you defined above
define service{
        use                     generic-service
        host_name               winserver
        service_description     NSClient++ Version
        check_command           check_nt!CLIENTVERSION
        }

# Create a service for monitoring the uptime of the server
# Change the host_name to match the name of the host you defined above
define service{
        use                     generic-service
        host_name               winserver
        service_description     Uptime
        check_command           check_nt!UPTIME
        }


# Create a service for monitoring CPU load
# Change the host_name to match the name of the host you defined above
define service{
        use                     generic-service
        host_name               winserver
        service_description     CPU Load
        check_command           check_nt!CPULOAD!-l 5,80,90
        }
....
....


I needed to register one more server. I wanted it to have same monitoring items with the previously registered server. How can I do for this? It is just simple. It has to use hostgroup. Let's modify configuration file like the following: 

$ vi /etc/nagios/objects/windows.cfg

## New host definition 
define host{
        use             windows-server  ; Inherit default values from a template
        host_name       adserver       ; The name we're giving to this host
        alias           Active Directory Server       ; A longer name
        address         192.168.20.168  ; IP address of the host
        }
....

## Members are added to the group
define hostgroup{
        hostgroup_name  windows-servers ; The name of the hostgroup
        alias           Windows Servers ; Long name of the group
        members         adserver, winserver     ; Comma separated list of hosts
        }
....

###################################################
###################################################
#
# SERVICE DEFINITIONS 
#
###################################################
###################################################

## hostgroup_name is added to "service definition"

define service{
        use                     generic-service
        host_name
        hostgroup_name          windows-servers
        service_description     NSClient++ Version
        check_command           check_nt!CLIENTVERSION
        }

....

With nagios online manual, the description for the "service definition" is:

hostgroup_name: This directive is used to specify the short name(s) of the hostgroup(s) that the service "runs" on or is associated with. The hostgroup_name may be used instead of, or in addition to, the host_name directive.

OR, there is another way to do this without using hostgroup: 

## Use a comma-separated list in the "service definition"

define service{
        use                     generic-service
        host_name               winserver, adserver
        hostgroup_name        
        service_description      Uptime
        check_command           check_nt!UPTIME
        }

You can also use a comma-separated list of hosts.

For the Linux host configuration file, it has the same way for this. In summarized, for the same monitoring service items with different hosts, it is effective way to manage by with the same hostgroup and define only specific sevices of each host.

Nagios: check_logfiles

Altibase is often called as hybrid database, which means store data both tablespace in a file system and memory. Normally, traditional database makes tablespace for storing data. But Altibase uses tablespace in memory. 
Memory is faster than Disk, this why we often store data into cache. 

Altimon is a monitoring deamon for this DB. Altimon periodically executes system queries which are written in configuration file. If Altimon detects one of results is over threshold, It writes error in another file. This post is for how I interact this log file with Nagios. 

1. What is check_logfiles ? 

check_logfiles is used to scan the lines of a file for regular expressions.

The plugin check_logfiles was designed to operate in mission critical environments where missing log lines after a logfile rotation could not be tolerated. 

When such a logfile rotation takes place, check_logfiles detects this and analyses the lines of the archived logfile, even if it's compressed.

It normally scans only the lines of a logfile which were added since the last run of the plugin.

The main features are:

- multiple regular expressions can be given

- expressions can be categorized as warning or critical

- it can handle any logfile rotation strategy

- hook scripts (either external scripts or a piece of perl-code in the configuration file) are possible, taking actions when a line matches a pattern. (for example, whenever a critical pattern is found, a nsca message is sent to the nagios server) 

2. Install check_logfiles & Test 

It is not able to install check_logfiles plugin via yum on CentOS. It hasn't been included its repository. It needs to download source code and install it manually. 

# Downloads check_logfiles

$ cd /downloads/

$ wget http://labs.consol.de/download/shinken-nagios-plugins/check_logfiles-3.5.1.tar.gz

$ ls -al

total 102490364

-rw-r--r--   1 root   root         138465 Dec 28  2007 check_logfiles-3.5.1.tar.gz

...

$ tar xzvf check_logfiles-3.5.1.tar.gz
$ cd check_logfiles-3.5.1

# The default directory of my nagios is  /usr/lib64/nagios/plugins/ 
$ ./configure --prefix=/usr/lib64/nagios/plugins/
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether make sets $(MAKE)... (cached) yes
variable with_seekfiles_dir is /var/tmp/check_logfiles
checking for sh... /bin/sh
checking for perl... /usr/bin/perl
checking for gzip... /bin/gzip
checking for gawk... /bin/gawk
checking for echo... /bin/echo
checking for sed... /bin/sed
checking for cat... /bin/cat
configure: creating ./config.status
config.status: creating Makefile
config.status: creating plugins-scripts/Makefile
config.status: creating plugins-scripts/subst
config.status: creating t/Makefile
                       --with-perl: /usr/bin/perl
                       --with-gzip: /bin/gzip
              --with-seekfiles-dir: /var/tmp/check_logfiles
              --with-protocols-dir: /tmp
               --with-trusted-path: /bin:/sbin:/usr/bin:/usr/sbin
                --with-nagios-user: nagios
               --with-nagios-group: nagios
                           
$ make
Making all in plugins-scripts
make[1]: Entering directory `/downloads/check_logfiles-3.5.1/plugins-scripts'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/downloads/check_logfiles-3.5.1/plugins-scripts'
Making all in t
make[1]: Entering directory `/downloads/check_logfiles-3.5.1/t'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/downloads/check_logfiles-3.5.1/t'
make[1]: Entering directory `/downloads/check_logfiles-3.5.1'
make[1]: Nothing to be done for `all-am'.
make[1]: Leaving directory `/downloads/check_logfiles-3.5.1'

$ make install
Making install in plugins-scripts
make[1]: Entering directory `/downloads/check_logfiles-3.5.1/plugins-scripts'
make[2]: Entering directory `/downloads/check_logfiles-3.5.1/plugins-scripts'
test -z "/usr/lib64/nagios/plugins/libexec" || mkdir -p -- "/usr/lib64/nagios/plugins/libexec"
/usr/bin/install -c 'check_logfiles' '/usr/lib64/nagios/plugins/libexec/check_logfiles'
make[2]: Nothing to be done for `install-data-am'.
make[2]: Leaving directory `/downloads/check_logfiles-3.5.1/plugins-scripts'
make[1]: Leaving directory `/downloads/check_logfiles-3.5.1/plugins-scripts'
Making install in t
make[1]: Entering directory `/downloads/check_logfiles-3.5.1/t'
make[2]: Entering directory `/downloads/check_logfiles-3.5.1/t'
make[2]: Nothing to be done for `install-exec-am'.
make[2]: Nothing to be done for `install-data-am'.
make[2]: Leaving directory `/downloads/check_logfiles-3.5.1/t'
make[1]: Leaving directory `/downloads/check_logfiles-3.5.1/t'
make[1]: Entering directory `/downloads/check_logfiles-3.5.1'
make[2]: Entering directory `/downloads/check_logfiles-3.5.1'
make[2]: Nothing to be done for `install-exec-am'.
make[2]: Nothing to be done for `install-data-am'.
make[2]: Leaving directory `/downloads/check_logfiles-3.5.1'
make[1]: Leaving directory `/downloads/check_logfiles-3.5.1'

$ ls -al /usr/lib64/nagios/plugins/libexec/
total 192
-rwxr-xr-x 1 root root 194274 Sep 20 15:26 check_logfiles

# Test check_files, I created file called test.log in /downloads and I added lines of words that 

# contained "ARLRM". 

$ /usr/lib64/nagios/plugins/libexec/check_logfiles --tag=altibase --logfile=/downloads/test.log --criticalpattern="ALARM"
CRITICAL - (4 errors in check_logfiles.protocol-2012-09-20-15-31-02) - [2012/06/12 00:00:12] [ALARM]:: [SESSION_COUNT.SID_COUNT] current [129] > checkValue [1] ...|altibase_lines=4 altibase_warnings=0 altibase_criticals=4 altibase_unknowns=0

# Next, added parameter "report=long", I wanted to see whole lines of errors.

/usr/lib64/nagios/plugins/libexec/check_logfiles --tag=altibase --logfile=/downloads/test.log --criticalpattern="ALARM" --report=long
CRITICAL - (6 errors in check_logfiles.protocol-2012-09-20-15-52-55) - [2012/06/12 11:38:22] [ALARM]:: [MEM_DATABASE_USE.ALLOC_MEM_MB] current [11296.09] > checkValue [7000] ...|altibase_lines=6 altibase_warnings=0 altibase_criticals=6 altibase_unknowns=0
tag altibase CRITICAL

[2012/06/12 00:00:11] [ALARM]:: [PROCESS.MEM_USAGE(KB)] Current (15456872) >= Limit (10240000)

[2012/06/12 00:00:12] [ALARM]:: [MEMSTAT_SUM.MAX_TOTAL_MB] current [17818.98] > checkValue [10240]

[2012/06/12 00:00:12] [ALARM]:: [MEM_DATABASE_USE.ALLOC_MEM_MB] current [11296.09] > checkValue [7000]

[2012/06/12 00:00:12] [ALARM]:: [SESSION_COUNT.SID_COUNT] current [129] > checkValue [1]
[2012/06/12 11:38:21] [ALARM]:: [MEMSTAT_SUM.MAX_TOTAL_MB] current [17866.01] > checkValue [10240]

[2012/06/12 11:38:22] [ALARM]:: [MEM_DATABASE_USE.ALLOC_MEM_MB] current [11296.09] > checkValue [7000]

Default parameters are follows: 

--tag= This is for Idenfication, If you want check one more combination of logfile/pattern, you should use this. (Optional) 

--logfile= log file for scan 

--criticalpattern= Regular expression for Critical 

--warningpattern= Regular expression for Warning  (Optional) 

--noprotocol=Switch off of logging match results in a separated file (default protocol files are created in /tmp

** Protocol – The matching lines can be written to a protocol file the name of which will be included in the plugin’s output. 

The path of protocol file definition can be defined in configuration file. The definitions in this file are written with Perl-syntax.

$protocolsdir: The default is /tmp or the directory which has been specified with the –with-protocol-dir of ./configure.

$protocolretention: The lifetime of protocol files in days. After these days the files are deleted automatically (default 7 days)

References: 
1. http://labs.consol.de/lang/en/nagios/check_logfiles/
2. http://exchange.nagios.org/directory/Plugins/Operating-Systems/Linux/check_logfiles/details

Nagios: Configuring passive check for Windows

Nagios' passive check has different way to active check. Active check starts from central nagios monitoring server, on the other hand, passive check begins at monitored server.

The above image(comes from http://www.nsclient.org) describes that how Nagios server and Windows computer communite each other. As you can see, this communication is triggered from monitored server.

Passive check is usually used for distributed monitoring. It can reduced the burden of Nagios server that preodically checked all of the monitored severs. In passive, Nagios sever has the role of simply getting monitoring data via NSCA (Nagios Server Check Acceptor).

On Monitored server (in this case, Windows server), in order to send data to Nagios server, I installed NSClient++ 0.4.x.(NSClient). NSClient support both check modes. Precisely speaking, however, it may have a bit different check commands. It usually use check_nt or check_nrpe in active and use check_nrpe in passive mode. While configuring NSClient, the most difficult thing was that config file had changed a lot between version 0.3.x and 0.4.x. First, config file name changed from nsc.ini to nsclient.ini. Secondly, changed expressions and propoperties in the config. When I was searched how to set properties in the config, most result was for 0.3.x.

About NSClient Configuration file :

1. Confugration general: http://www.nsclient.org/nscp/wiki/doc/configuration:
2. 0.4.x config file: http://www.nsclient.org/nscp/wiki/doc/configuration/0.4.x
3. Example for 0.4.x: http://nsclient.org/nscp/discussion/message/2575



Enabling Passive Check ( Nagios Server)
$ vi /etc/nagios/nagios.cfg
# PASSIVE SERVICE CHECK ACCEPTANCE OPTION
# This determines whether or not Nagios will accept passive
# service checks results when it initially (re)starts.
# Values: 1 = accept passive checks, 0 = reject passive checks
accept_passive_service_checks=1

$ nagios -v /etc/nagios/nagios.cfg
.....
Total Warnings: 0
Total Errors: 0

# Installing NSCA (Nagios Server Check Acceptor)
$ yum install nsca

# Start NSCA
$ service nsca start

# check if specified port (it left as default 5667) is open
$ netstat -na | grep 5667tcp 0 0 0.0.0.0:5667 0.0.0.0:* LISTEN


Configuring nsclient.ini ( Windows Server)
Here was my C:\Program Files\NSClient++sclient.ini.
===============================

; Undocumented section
[/modules]

; CheckDisk - CheckDisk can check various file and disk related things.

; The current version has commands to check Size of hard drives and directories.
CheckDisk = 1

; Event log Checker. - Check for errors and warnings in the event log.

; This is only supported through NRPE so if you plan to use only NSClient this wont help you at all.
CheckEventLog = 1

; Check External Scripts - A simple wrapper to run external scripts and batch files.
CheckExternalScripts = 1

; Helper function - Various helper function to extend other checks. This is also only supported through NRPE.
CheckHelpers = 1

; Check NSCP - Checkes the state of the agent
CheckNSCP = 1

; CheckSystem - Various system related checks, such as CPU load, process state, service state memory usage and PDH counters.
CheckSystem = 1

; CheckWMI - CheckWMI can check various file and disk related things.

; The current version has commands to check Size of hard drives and directories.
CheckWMI = 1

; NRPE server - A simple server that listens for incoming NRPE connection and handles them.

; NRPE is preferred over NSClient as it is more flexible. You can of cource use both NSClient and NRPE.
NRPEServer = 1

; NSCAClient - Passive check support (needs NSCA on nagios server).

; Avalible crypto are: {0=No Encryption (not safe), 1=XOR, 2=DES, 3=DES-EDE3, 4=CAST-128, 6=XTEA,

; 8=Blowfish, 9=Twofish, 11=RC2, 14=AES, 15=AES, 16=AES, 20=Serpent, 23=GOST}
NSCAClient = 1

; NSClient server - A simple server that listens for incoming NSClient (check_nt) connection and handles them.

; Although NRPE is the preferred method NSClient is fully supported and can be used for simplicity or for compatibility.
NSClientServer = 1

# Scheduler
# A scheduler which schedules checks at regular intervals
Scheduler=1

; Undocumented section
[/settings/default]

; ALLOWED HOSTS - A comaseparated list of allowed hosts. You can use netmasks (/ syntax) or * to create ranges.
allowed hosts = xxx.xxx.xxx.xxx

; PASSWORD - Password used to authenticate againast server
password = xxxxxxxx

; Section for NRPE (NRPEListener.dll) (check_nrpe) protocol options.
[/settings/NRPE/server]

; Section for NSCA passive check module.
[/settings/NSCA/client]

# NSCA CLIENT SECTION
# Section for NSCA passive check module.
[/settings/NSCA/client]
# DELAY
#
delay=0
# HOSTNAME
# The host name of this host if set to blank (default) the windows name of the computer will be used.
hostname=adserver
# CHANNEL
# The channel to listen to.
channel=NSCA

; Target definition for: default
[/settings/NSCA/client/targets/default]

;# ENCRYPTION METHOD
; This option determines the method by which the send_nsca client will encrypt the packets it sends
; to the nsca daemon. The encryption method you choose will be a balance between security and
; performance, as strong encryption methods consume more processor resources.
; You should evaluate your security needs when choosing an encryption method.
;
; Note: The encryption method you specify here must match the decryption method the nsca daemon uses
; (as specified in the nsca.cfg file)!!
; Values:
; 0 = None (Do NOT use this option)
; 1 = Simple XOR (No security, just obfuscation, but very fast)
; 2 = DES
; 3 = 3DES (Triple DES)
; 4 = CAST-128
; 6 = xTEA
; 8 = BLOWFISH
; 9 = TWOFISH
; 11 = RC2
; 14 = RIJNDAEL-128 (AES)
; 20 = SERPENT
encryption=1

;
;# ENCRYPTION PASSWORD
; This is the password/passphrase that should be used to encrypt the sent packets.
password=xxxxxxxxxx
;
;# NAGIOS SERVER ADDRESS
; The address to the nagios server to submit results to.
;host=

;# TARGET ADDRESS
;# Target host address
address=xxx.xxx.xxx.xxx

;# TARGET PORT
;# The target server port
port=5667

# TIMEOUT
# Timeout when reading/writing packets to/from sockets.
timeout=30

; A list of aliases available. An alias is an internal command that has been "wrapped" (to add arguments). Be careful so you don't create loops (ie check_loop=check_a, check_a=check_loop)
[/settings/external scripts/alias]
alias_cpu=checkCPU warn=80 crit=90 time=5m time=1m time=30s

; Configure log properties.
[/settings/log]
;# LOG DEBUG
; Set to 1 if you want debug message printed in the log file (debug messages are always printed to stdout when run with -test)
debug=1
;# LOG DATE MASK
; The format to for the date/time part of the log entry written to file.
;date_mask=%Y-%m-%d %H:%M:%S

; Configure log file properties.
[/settings/log/file]
file=C:\Program Files\NSClient++\NSC.log

; Section for configuring the shared session.
[/settings/shared session]

; A list of avalible remote target systems
[/settings/targets]

[/settings/scheduler/schedules/default]
channel=NSCA
interval=30s
report=all
[/settings/scheduler/schedules]
CPU Load=alias_cpu

Then, NClient service has to be started OR

can stat as test mode

C:\Users\Administrator\Downloads\NSCP>nscp test
d vice\logger_impl.cpp:371 Creating logger: console


d rvice\NSClient++.cpp:374 NSClient++ 0,4,0,172 2012-05-08 x64 Loading settings and logger...


d ngs_manager_impl.cpp:170 No entries found looking in (adding default): C:/Users/Administrator/Downloads/NSCP//boot.ini


d ngs_manager_impl.cpp:179 Boot order: old://${exe-path}/nsc.ini, ini://${shared-path}/nsclient.ini


d ngs_manager_impl.cpp:193 No valid settings found (tried): old://${exe-path}/nsc.ini, ini://${shared-path}/nsclient.ini


e ngs_manager_impl.cpp:201 Settings contexts exausted, will create a new ini://${shared-path}/nsclient.ini


d ngs_manager_impl.cpp:73 Creating instance for: ini://${shared-path}/nsclient.ini


d ngs/settings_ini.hpp:268 Reading INI settings from: C:/Users/Administrator/Downloads/NSCP//nsclient.ini


l rvice\NSClient++.cpp:385 NSClient++ 0,4,0,172 2012-05-08 x64 booting...


d rvice\NSClient++.cpp:386 Booted settings subsystem...


d rvice\NSClient++.cpp:453 On crash: restart: NSClientpp


d rvice\NSClient++.cpp:465 Archiving crash dumps in: C:/Users/Administrator/Downloads/NSCP//crash-dumps


d rvice\NSClient++.cpp:532 booting::loading plugins


d rvice\NSClient++.cpp:604 NSClient++ - 0,4,0,172 2012-05-08 Started!


l ce\simple_client.hpp:32 Enter command to inject or exit to terminate..

Defining service ( Nagios Server)
$ vi /etc/nagios/object/nagios.cfg

define host{
use windows-server ; Inherit default values from a template
host_name adserver ; The name we're giving to this host
alias Active Directory Server ; A longer name associated with the host
passive_checks_enabled 1
active_checks_enabled 0
address xxx.xxx.xxx.xxx ; IP address of the host
}

define service{
use generic-service
host_name adserver
passive_checks_enabled 1
active_checks_enabled 0
service_description CPU Load
check_command check_nrpe!alias_cpu }

$ vi /etc/nagios/object/commands.cfg
# 'check_nrpe' command definition
define command{
command_name check_nrpe
command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}


$ nagios -v /etc/nagios/nagios.cfg
Total Warnings: 0Total Errors: 0

$ servicee nagios restart

$ tail /var/log/nagios/nagios.log
[1342862307] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;adserver;cpu_check;0;OK CPU Load ok.|'5m'=0%;80;90 '1m'=1%;80;90 '30s'=2%;80;90
[1342862307] Warning: Passive check result was received for service 'cpu_check' on host 'adserver', but the service could not be found! ==> This warning casued not to defined cpu_check in commands.cfg
[1342862795] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;adserver;CPU Load;0;OK CPU Load ok.|'5m'=0%;80;90 '1m'=1%;80;90 '30s'=3%;80;90
[1342862805] PASSIVE SERVICE CHECK: adserver;CPU Load;0;OK CPU Load ok.
[1342862825] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;adserver;CPU Load;0;OK CPU Load ok.|'5m'=0%;80;90 '1m'=2%;80;90 '30s'=2%;80;90
[1342862835] PASSIVE SERVICE CHECK: adserver;CPU Load;0;OK CPU Load ok.
[1342862855] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;adserver;CPU Load;0;OK CPU Load ok.|'5m'=0%;80;90 '1m'=2%;80;90 '30s'=2%;80;90
[1342862865] PASSIVE SERVICE CHECK: adserver;CPU Load;0;OK CPU Load ok.


I am being nervous and impatient when I coudn't find effient answers to my problem. Espeically, It could be deeper during a project. It is like a situcation where time can be very limited and another works are waiting for me. Writing articles in this blog mean to save my time when I encounted in a similar problem and hope these save serchers' time.


Reference sites:
1. http://www.nsclient.org/nscp/wiki/doc/usage/nagios/nsca
2. http://assets.nagios.com/downloads/nagiosxi/docs/Using_NSClient_For_Passive_Checks.pdf

Posted in Blogsy (iPad)