Monitoring: Setting up Nagois + ndoutils

Nagios is one of the great tool that monitoring servers. The benefit of this is that it has many plugins which supports various software and hardware. Here, ndoutils is a plugin which stores status data comes from nagios into databases like Mysql or PostgreSQL.

Ndoutils has two components. The one is ndomod. nodomod has role of exporting nagios runtime data from nagios daemon and then write it to a file or unix socket or  tcp. The other is ndo2db, which is sending nagios data from ndomod via unix socket or tcp to database.

For more detailed information about how ndoutils works, refer to http://www.nagios-wiki.de/nagios/ndo/start (Written in German, you can translate this on google translate)

How to install and setting
# Download and configure to install ndoutils
$ wget http://sourceforge.net/projects/nagios/files/ndoutils-1.x/ndoutils-1.5.2/ndoutils-1.5.2.tar.gz/download$ tar -xzvf ndoutils-1.5.2.tar.gz$ cd ndoutils-1.5.2$ ./configure --with-mysql-lib=/usr/lib64/mysql --prefix=/usr/local/ndoutils-1.5.2 --with-ndo2db-user=nagios --with-ndo2db-group=nagios
configure: WARNING: unrecognized options: --with-mysql-lib
....
hecking for mysql_config... /usr/bin/mysql_config
checking for mysql_init in -lmysqlclient... yes
MySQL library and include file(s) were found!
....
*** Configuration summary for ndoutils 1.5.2 06-08-2012 ***:
General Options:
-------------------------
NDO2DB user:    nagios
NDO2DB group:   nagios
Review the options above for accuracy.  If they look okay,
type 'make' to compile the NDO utilities.
$ make
$ make install
Hint: NDOUtils Installation against Nagios v3.x
  completed.
  If you want to install NDOUtils for Nagios v2.x
  please type  'make install-2x
  Next step should be the database initialization/upgrade
  cd into the db/ directory and either:
     ./installdb  (for a new installation) or:
     ./upgradedb  (for an existing one)
make[1]: Leaving directory `/downloads/ndoutils-1.5.2/src'
Main NDOUtils components installed


# Creating database and user
$ mysql -u root -p mysql
mysql> create database nagios_db;
Query OK, 1 row affected (0.00 sec)
mysql> create user nagios@'localhost' identified by 'nagios'; Query OK, 0 rows affected (0.00 sec)
mysql> create user nagios@'%' identified by 'nagios'; Query OK, 0 rows affected (0.00 sec)
mysql> grant all on nagios_db.* to nagios@'localhost';
Query OK, 0 rows affected (0.00 sec)
mysql> grant all on nagios_db.* to nagios@'%';
Query OK, 0 rows affected (0.00 sec)
mysql> exit;


# Import ndo tables into mysql database 
$ cd db
./installdb -u nagios -p nagios -h localhost -d nagios_db
DBD::mysql::db do failed: Table 'nagios_db.nagios_dbversion' doesn't exist at ./installdb line 51.
** Creating tables for version 1.5.2
     Using mysql.sql for installation...
** Updating table nagios_dbversion
Done!


# var directory (lock file will be created here) 
$ cd /usr/local/ndoutils-1.5.2/
$ mkdir ./var
$ sudo chown -R nagios:nagios ./var


# Copy config files into Nagios directory and change setting values 
$ cp /downlaods/ndoutils-1.5.2/config/ndomod.cfg-sample /etc/nagios/ndomod.cfg
$ cp /downlaods/ndoutils-1.5.2config/ndo2db.cfg-sample /etc/nagios/ndo2db.cfg
$ chown nagios:nagios /etc/nagios/ndo*.cfg
$ ls -al /etc/nagios/ndo*
-rw-rw-rw- 1 nagios nagios 4516 Jul 12 11:38 /etc/nagios/ndo2db.cfg
-rw-rw-rw- 1 nagios nagios 5154 Jul 12 11:39 /etc/nagios/ndomod.cfg


# This configuration directive will load the ndomod.o NEB module when Nagios is started. You will need to restart Nagios to make the module active
$ vi /etc/nagios/nagios.cfg
broker_module=/usr/local/ndoutils-1.5.2/bin/ndomod.o config_file=/etc/nagios/ndomod.cfg  


# Correct database name, user, password
$ vi /etc/nagios/ndo2db.cfg
ndo2db_user=nagios 
ndo2db_group=nagios
db_host=localhost
db_name=nagios_db
db_user=nagios 
db_pass=*****


# Register ndo2db as daemon
$ cp /downloads/ndoutils-1.5.2/daemon-init /etc/init.d/ndo2db
$ vi /etc/init.d/ndo2db
servicename=ndo2db 
prefix=/usr/local/ndoutils-1.5.2 
exec_prefix=/usr/local/ndoutils-1.5.2 
Ndo2dbBin=/usr/local/ndoutils-1.5.2/bin/ndo2db
Ndo2dbCfgFile=/etc/nagios/ndo2db.cfg 
Ndo2dbVarDir=/usr/local/ndoutils-1.5.2/var Ndo2dbRunFile=$Ndo2dbVarDir/ndo2db.lock 
Ndo2dbLockDir=/var/lock/subsys 
Ndo2dbLockFile=ndo2db 
Ndo2dbUser=nagios 
Ndo2dbGroup=nagios


# Start NDO first and Nagios
$ su nagios
$ sudo /etc/init.d/ndo2db start
$ sudo /etc/init.d/nagios start
$ sudo ps aux | grep ndo2db
nagios   20109  0.0  0.0  49780   616 ?        Ss   11:39   0:00 /usr/local/ndoutils-1.5.2/bin/ndo2db -c /etc/nagios/ndo2db.cfg
nagios   20123  0.0  0.0  49780  1372 ?        S    11:39   0:01 /usr/local/ndoutils-1.5.2/bin/ndo2db -c /etc/nagios/ndo2db.cfg
nagios   20124  0.0  0.0  49784  1540 ?        S    11:39   0:02 /usr/local/ndoutils-1.5.2/bin/ndo2db -c /etc/nagios/ndo2db.cfg


# check log file
$ sudo tail -f /var/log/messages
Jul 12 10:58:26 cloudmanager nagios: Successfully shutdown... (PID=16525)
Jul 12 10:58:26 cloudmanager nagios: ndomod: Shutdown complete.
Jul 12 10:58:26 cloudmanager nagios: Event broker module '/usr/local/ndoutils-1.5.2/bin/ndomod.o' deinitialized successfully.
Jul 12 11:39:24 cloudmanager nagios: Nagios 3.3.1 starting... (PID=20122)
Jul 12 11:39:24 cloudmanager nagios: Local time is Thu Jul 12 11:39:24 KST 2012
Jul 12 11:39:24 cloudmanager nagios: LOG VERSION: 2.0 
Jul 12 11:39:24 cloudmanager nagios: ndomod: NDOMOD 1.5.2 (06-08-2012) Copyright (c) 2009 Nagios Core Development Team and Community Contributors 
Jul 12 11:39:24 cloudmanager nagios: ndomod: Successfully connected to data sink.  0 queued items to flush.    => success log 
Jul 12 11:39:24 cloudmanager nagios: Event broker module '/usr/local/ndoutils-1.5.2/bin/ndomod.o' initialized successfully.
Jul 12 11:39:24 cloudmanager nagios: Finished daemonizing... (New PID=20127)


# check database 
$ mysql -u nagios -p nagios_db
mysql> select count(*) from nagios_conninfo;
+----------+
| count(*) |
+----------+
|        1 |
+----------+
1 row in set (0.01 sec)

mysql> select * from nagios_instances;
+-------------+---------------+----------------------+
| instance_id | instance_name | instance_description |
+-------------+---------------+----------------------+
|           1 | default       |                      |
+-------------+---------------+----------------------+
1 row in set (0.00 sec)
mysql> quit;

These are two tables that ndo2db put initial data into.

select * from nagios_conninfo;
select * from nagios_instances;

Troubleshooting
1. Error: Support for the specified database server is either not yet supported, or was not found on your system.


$ /usr/local/ndoutils-1.5.2/bin/ndo2db -c /etc/nagios/ndo2db.cfg
Support for the specified database server is either not yet supported, or was not found on your system.


$ vi /var/log/nagios/nagios.log
[1341910636] ndomod: Could not open data sink!  I'll keep trying, but some output may get lost... 
[1341910636] Event broker module '/usr/local/ndoutils-1.5.2/bin/ndomod.o' initialized successfully.........
[1341911548] ndomod: Still unable to connect to data sink.  456 items lost, 5000 queued items to flush.


Solution: you must install mysql-devel

$ yum install mysql-devel


# checking whether the following message during ./configure, If you find it, you have been installed mysql-devel on your server.
$ ./configure --enable-mysql --with-mysql-lib=/usr/lib64/mysql --prefix=/usr/local/ndoutils-1.5.2
......
checking for mysql_config... /usr/bin/mysql_config 
checking for mysql_init in -lmysqlclient... yes
MySQL library and include file(s) were found!
......
......


2. Error: max retries exceeded sending message to queue. Kernel queue parameters may neeed to be tuned. See README
$ tail -f /var/log/messages
Jul 11 19:45:01 ndo2db: Warning: Retrying message send. This can occur because you have too few messages allowed or too few total bytes allowed in message queues. You are currently using 128 of 15424 messaged and 131072 of 131072 bytes in the queue. See REAMDE for kernel tuning options
Jul 11 19:45:01 ndo2db: Warning: queue send error, retrying...
Jul 11 19:45:01 ndo2db: Error: max retries exceeded sending message to queue. Kernel queue parameters may neeed to be tuned. See README.

Solution: raise up number in msgmax and msgmnb
$ echo 131072000 > /proc/sys/kernel/msgmax 
$ echo 131072000 > /proc/sys/kernel/msgmnb
# Type the same number in sysctl.conf to apply permanently.
$ vi /etc/sysctl.conf


3. Error: Can't populate data into database. but it said that it worked fine in the log file

Solution: I recommend that you install NDOutil again. I installed ndoutils and configured well. my log said that it worked well. However, it failed inserting nagios data into database.

I spent two days for this setting up. I stuck up the above problems during the setting. Especially. The error NO. 3 was made me the most confused. If you fell in the problem like NO. 3, it would be the best solution to install again. I was googling about this and find some results which indicated it worked after they installed ndoutils agin. Hope this help.

-------
Added on 2012-12-28

4. Error: Could not bind socket: Address already in use

Solution: remove its lock file. (Got an idea from http://blog.connexeon.com/sysadmin/linux/nagios-ndo2db-could-not-bind-socket/) 

$ find / -name ndo.sock -print
/usr/local/ndoutils-1.5.2/var/ndo.sock
$ rm /usr/local/ndoutils-1.5.2/var/ndo.sock
rm: remove socket `/usr/local/ndoutils-1.5.2/var/ndo.sock'? y
$ service ndo2db start
Starting ndo2db: done.



Reference sites: 
1. http://onaxer.com/blog/blog/2010/03/05/imports-historical-nagios-log-files-into-the-mysql-database/
2. http://www.r71.nl/kb/technical/363-ndo2db-and-kernel-parameters

3. http://sourceforge.net/mailarchive/forum.php?thread_name=B5224EE8-E51A-4E14-BA61-B9B6418FDD09%40zarzax.com&forum_name=nagios-users

4. http://www.nagios-wiki.de/nagios/ndo/start