2012년 7월 12일 목요일

Monitoring: Setting up Nagois + ndoutils


Nagios is one of the great tool that monitoring servers. The benefit of this is that it has many plugins which supports various software and hardware. Here, ndoutils is a plugin which stores status data comes from nagios into databases like Mysql or PostgreSQL.

Ndoutils has two components. The one is ndomod. nodomod has role of exporting nagios runtime data from nagios daemon and then write it to a file or unix socket or  tcp. The other is ndo2db, which is sending nagios data from ndomod via unix socket or tcp to database.

For more detailed information about how ndoutils works, refer to http://www.nagios-wiki.de/nagios/ndo/start (Written in German, you can translate this on google translate)

How to install and setting
# Download and configure to install ndoutils
$ wget http://sourceforge.net/projects/nagios/files/ndoutils-1.x/ndoutils-1.5.2/ndoutils-1.5.2.tar.gz/download$ tar -xzvf ndoutils-1.5.2.tar.gz$ cd ndoutils-1.5.2$ ./configure --with-mysql-lib=/usr/lib64/mysql --prefix=/usr/local/ndoutils-1.5.2 --with-ndo2db-user=nagios --with-ndo2db-group=nagios
configure: WARNING: unrecognized options: --with-mysql-lib
....
hecking for mysql_config... /usr/bin/mysql_config
checking for mysql_init in -lmysqlclient... yes
MySQL library and include file(s) were found!
....
*** Configuration summary for ndoutils 1.5.2 06-08-2012 ***:
General Options:
-------------------------
NDO2DB user:    nagios
NDO2DB group:   nagios
Review the options above for accuracy.  If they look okay,
type 'make' to compile the NDO utilities.
$ make
$ make install
Hint: NDOUtils Installation against Nagios v3.x
  completed.
  If you want to install NDOUtils for Nagios v2.x
  please type  'make install-2x
  Next step should be the database initialization/upgrade
  cd into the db/ directory and either:
     ./installdb  (for a new installation) or:
     ./upgradedb  (for an existing one)
make[1]: Leaving directory `/downloads/ndoutils-1.5.2/src'
Main NDOUtils components installed


# Creating database and user
$ mysql -u root -p mysql
mysql> create database nagios_db;
Query OK, 1 row affected (0.00 sec)
mysql> create user nagios@'localhost' identified by 'nagios'; Query OK, 0 rows affected (0.00 sec)
mysql> create user nagios@'%' identified by 'nagios'; Query OK, 0 rows affected (0.00 sec)
mysql> grant all on nagios_db.* to nagios@'localhost';
Query OK, 0 rows affected (0.00 sec)
mysql> grant all on nagios_db.* to nagios@'%';
Query OK, 0 rows affected (0.00 sec)
mysql> exit;


# Import ndo tables into mysql database 
$ cd db
./installdb -u nagios -p nagios -h localhost -d nagios_db
DBD::mysql::db do failed: Table 'nagios_db.nagios_dbversion' doesn't exist at ./installdb line 51.
** Creating tables for version 1.5.2
     Using mysql.sql for installation...
** Updating table nagios_dbversion
Done!


# var directory (lock file will be created here) 
$ cd /usr/local/ndoutils-1.5.2/
$ mkdir ./var
$ sudo chown -R nagios:nagios ./var


# Copy config files into Nagios directory and change setting values 
$ cp /downlaods/ndoutils-1.5.2/config/ndomod.cfg-sample /etc/nagios/ndomod.cfg
$ cp /downlaods/ndoutils-1.5.2config/ndo2db
.cfg-sample /etc/nagios/ndo2db.cfg
$ chown nagios:nagios /etc/nagios/ndo*.cfg
$ ls -al /etc/nagios/ndo*
-rw-rw-rw- 1 nagios nagios 4516 Jul 12 11:38 /etc/nagios/ndo2db.cfg
-rw-rw-rw- 1 nagios nagios 5154 Jul 12 11:39 /etc/nagios/ndomod.cfg


# This configuration directive will load the ndomod.o NEB module when Nagios is started. You will need to restart Nagios to make the module active
$ vi /etc/nagios/nagios.cfg
broker_module=/usr/local/ndoutils-1.5.2/bin/ndomod.o config_file=/etc/nagios/ndomod.cfg  


# Correct database name, user, password
$ vi /etc/nagios/ndo2db.cfg
ndo2db_user=nagios 

ndo2db_group=nagios
db_host=localhost
db_name=nagios_db
db_user=nagios 
db_pass=*****


# Register ndo2db as daemon
$ cp /downloads/ndoutils-1.5.2/daemon-init /etc/init.d/ndo2db
$ vi /etc/init.d/ndo2db
servicename=ndo2db 

prefix=/usr/local/ndoutils-1.5.2 
exec_prefix=/usr/local/ndoutils-1.5.2 
Ndo2dbBin=/usr/local/ndoutils-1.5.2/bin/ndo2db
Ndo2dbCfgFile=/etc/nagios/ndo2db.cfg 
Ndo2dbVarDir=/usr/local/ndoutils-1.5.2/var Ndo2dbRunFile=$Ndo2dbVarDir/ndo2db.lock 
Ndo2dbLockDir=/var/lock/subsys 
Ndo2dbLockFile=ndo2db 
Ndo2dbUser=nagios 
Ndo2dbGroup=nagios


# Start NDO first and Nagios
$ su nagios
$ sudo /etc/init.d/ndo2db start
$ sudo /etc/init.d/nagios start
$ sudo ps aux | grep ndo2db
nagios   20109  0.0  0.0  49780   616 ?        Ss   11:39   0:00 /usr/local/ndoutils-1.5.2/bin/ndo2db -c /etc/nagios/ndo2db.cfg
nagios   20123  0.0  0.0  49780  1372 ?        S    11:39   0:01 /usr/local/ndoutils-1.5.2/bin/ndo2db -c /etc/nagios/ndo2db.cfg
nagios   20124  0.0  0.0  49784  1540 ?        S    11:39   0:02 /usr/local/ndoutils-1.5.2/bin/ndo2db -c /etc/nagios/ndo2db.cfg


# check log file
$ sudo tail -f /var/log/messages
Jul 12 10:58:26 cloudmanager nagios: Successfully shutdown... (PID=16525)
Jul 12 10:58:26 cloudmanager nagios: ndomod: Shutdown complete.
Jul 12 10:58:26 cloudmanager nagios: Event broker module '/usr/local/ndoutils-1.5.2/bin/ndomod.o' deinitialized successfully.
Jul 12 11:39:24 cloudmanager nagios: Nagios 3.3.1 starting... (PID=20122)
Jul 12 11:39:24 cloudmanager nagios: Local time is Thu Jul 12 11:39:24 KST 2012
Jul 12 11:39:24 cloudmanager nagios: LOG VERSION: 2.0 

Jul 12 11:39:24 cloudmanager nagios: ndomod: NDOMOD 1.5.2 (06-08-2012) Copyright (c) 2009 Nagios Core Development Team and Community Contributors 
Jul 12 11:39:24 cloudmanager nagios: ndomod: Successfully connected to data sink.  0 queued items to flush.    => success log 
Jul 12 11:39:24 cloudmanager nagios: Event broker module '/usr/local/ndoutils-1.5.2/bin/ndomod.o' initialized successfully.
Jul 12 11:39:24 cloudmanager nagios: Finished daemonizing... (New PID=20127)


# check database 
$ mysql -u nagios -p nagios_db
mysql> select count(*) from nagios_conninfo;

+----------+
| count(*) |
+----------+
|        1 |
+----------+
1 row in set (0.01 sec)

mysql> select * from nagios_instances;
+-------------+---------------+----------------------+
| instance_id | instance_name | instance_description |
+-------------+---------------+----------------------+
|           1 | default       |                      |
+-------------+---------------+----------------------+
1 row in set (0.00 sec)
mysql> quit;

These are two tables that ndo2db put initial data into.


select * from nagios_conninfo;
select * from nagios_instances;

Troubleshooting
1. Error: Support for the specified database server is either not yet supported, or was not found on your system.


$ /usr/local/ndoutils-1.5.2/bin/ndo2db -c /etc/nagios/ndo2db.cfg
Support for the specified database server is either not yet supported, or was not found on your system.


$ vi /var/log/nagios/nagios.log
[1341910636] ndomod: Could not open data sink!  I'll keep trying, but some output may get lost... 
[1341910636] Event broker module '/usr/local/ndoutils-1.5.2/bin/ndomod.o' initialized successfully.........
[1341911548] ndomod: Still unable to connect to data sink.  456 items lost, 5000 queued items to flush.


Solution: you must install mysql-devel

$ yum install mysql-devel


# checking whether the following message during ./configure, If you find it, you have been installed mysql-devel on your server.
$ ./configure --enable-mysql --with-mysql-lib=/usr/lib64/mysql --prefix=/usr/local/ndoutils-1.5.2
......
checking for mysql_config... /usr/bin/mysql_config 
checking for mysql_init in -lmysqlclient... yes
MySQL library and include file(s) were found!
......

......


2. Error: max retries exceeded sending message to queue. Kernel queue parameters may neeed to be tuned. See README

$ tail -f /var/log/messages
Jul 11 19:45:01 ndo2db: Warning: Retrying message send. This can occur because you have too few messages allowed or too few total bytes allowed in message queues. You are currently using 128 of 15424 messaged and 131072 of 131072 bytes in the queue. See REAMDE for kernel tuning options
Jul 11 19:45:01 ndo2db: Warning: queue send error, retrying...
Jul 11 19:45:01 ndo2db: Error: max retries exceeded sending message to queue. Kernel queue parameters may neeed to be tuned. See README.

Solution: raise up number in msgmax and msgmnb
$ echo 131072000 > /proc/sys/kernel/msgmax 

$ echo 131072000 > /proc/sys/kernel/msgmnb
# Type the same number in sysctl.conf to apply permanently.
$ vi /etc/sysctl.conf



3. Error: Can't populate data into database. but it said that it worked fine in the log file

Solution: I recommend that you install NDOutil again. I installed ndoutils and configured well. my log said that it worked well. However, it failed inserting nagios data into database.

I spent two days for this setting up. I stuck up the above problems during the setting. Especially. The error NO. 3 was made me the most confused. If you fell in the problem like NO. 3, it would be the best solution to install again. I was googling about this and find some results which indicated it worked after they installed ndoutils agin. Hope this help.

-------
Added on 2012-12-28

4. Error: Could not bind socket: Address already in use

Solution: remove its lock file. (Got an idea from http://blog.connexeon.com/sysadmin/linux/nagios-ndo2db-could-not-bind-socket/

$ find / -name ndo.sock -print
/usr/local/ndoutils-1.5.2/var/ndo.sock
$ rm /usr/local/ndoutils-1.5.2/var/ndo.sock
rm: remove socket `/usr/local/ndoutils-1.5.2/var/ndo.sock'? y
$ service ndo2db start
Starting ndo2db: done.



Reference sites: 
1. http://onaxer.com/blog/blog/2010/03/05/imports-historical-nagios-log-files-into-the-mysql-database/
2. http://www.r71.nl/kb/technical/363-ndo2db-and-kernel-parameters

댓글 38개:

  1. I'm just curious, are you Korean? I'm Korean.

    답글삭제
  2. Sure, I'm a Korean too. What made you curios?

    답글삭제
  3. Many Thanks, this worked as breeze.

    Just one mistake.

    $ cp /downlaods/ndoutils-1.5.2/config/ndomod.cfg-sample /etc/nagios/ndomod.cfg
    $ cp /downlaods/ndoutils-1.5.2config/ndo2db.cfg-sample /etc/nagios/ndo2db.cfg

    Here the second line should have ndo2db.cfg-sample.

    답글삭제
  4. Thanks for correcting. It seemed that it misspelled it while I was writing.

    답글삭제
  5. Dear Yeonki,

    i couldn't find these two files:

    $ cp /downlaods/ndoutils-1.5.2/config/ndomod.cfg-sample /etc/nagios/ndomod.cfg
    $ cp /downlaods/ndoutils-1.5.2config/ndo2db.cfg-sample /etc/nagios/ndo2db.cfg

    is there anything wrong on my installation?

    regards,
    riza

    답글삭제
  6. Hi, riza.
    Sorry to misspell.

    Firs of all, Have you downloaded ndoutils-1.5.2.tar.gz and untar this file? or if you are already did it, What directory did you untar ndoutils-1.5.2.tar.gz at ?

    Those cfg files are located into ["your ndoutils-1.5.2"]/config/.

    In my case, mines are placed in /downloads/ndoutils-1.5.2/config/

    Thank you.

    답글삭제
  7. mine are on:

    root@netwatch:~/ndoutils-1.5.2#

    its not on download folder. geez im new in linux :p

    please tell me what folder is my installation?

    thank you

    답글삭제
  8. i've just do this:

    root@netwatch:~/ndoutils-1.5.2/config# cp ndomod.cfg-sample /etc/nagios3/ndomod.cfg

    답글삭제
  9. now i've got no instance running after

    mysql> select * from nagios_instances;
    Empty set (0.00 sec)

    is it okay?

    답글삭제
  10. It seems something wrong.
    There must be at list one record in the table.
    Have you checked log file?
    Please look at /var/log/nagios/nagios.log or /usr/local/nagios/var/nagios.log.

    답글삭제
  11. hi!
    thanks for this help :D
    I've the 3rd problem...some solutions?
    thankx

    답글삭제
  12. Hi,
    Unfortunately, there isn't quite clear answer to this 3rd problem. As I mentioned above, It would be a good solution to install it again. I installed it again and then it worked well. But I didn't know exactly what is was wrong.

    Cheers.

    답글삭제
  13. Nothing to do :(
    should be a mysql problem?

    답글삭제
  14. Hi, Alessandro,
    After you installed it again, didn't work it?

    답글삭제
  15. 작성자가 댓글을 삭제했습니다.

    답글삭제
  16. Dear Yeonki,
    Thanks for your post.It is very useful.

    But I've got this error log in /etc/log/messages

    ndomod: Could not open data sink! I'll keep trying, but some output may get lost...

    I've already installed mysql-devel.
    And I checked nagios.cfg, ndo2db.cfg and ndomod.cfg.
    Can you help me?

    답글삭제
  17. It might be permission issue according to http://forums.meulie.net/viewtopic.php?f=59&t=4406.
    Can you change permission of ndo2db.cfg and ndomod.cfg? (chmod 666 ndo2db.cfg ndomod.cfg)

    답글삭제
    답글
    1. Dear Yeonki,
      Thank-you so much!!
      It's working good.

      Have a really nice day!

      삭제
    2. Dear Yeonki,

      Can you help me again please?
      I have to redeploy to a remote db server

      So, I've modifed the ndo2db.cfg like below

      socket_type=tcp
      db_host=remote db server IP

      And I've modifed the ndomod.cfg like below

      output_type=tcpsocket
      output=remote db server IP

      And I've installed mysql(include mysql-devel) and ndo2db on the remote db server.
      I've also opened 5668 port and 3306 port on the remote db server.


      But, I've got [ndomod: Could not open data sink! I'll keep trying, but some output may get lost... ] error log again.

      Do you know what I have to do anything?

      삭제
    3. Mysql server would be okay to install on a remote server. You can set ip of mysql in configuration file.
      In my opinion, however, Ndo2db is different. I recommend you that you should install nagios and ndo2db on the same server.

      삭제
    4. Thank-you for your help and your recommendation.
      Have a nice day!



      삭제
  18. Hi,
    I am getting the following error in /var/log/messages
    Oct 17 11:01:06 seldmgte58 ndo2db: mysql_error: 'Unknown column 'long_output' in 'field list''

    답글삭제
    답글
    1. Issue fixed after run the following SQL commands
      ALTER TABLE nagios_eventhandlers ADD long_output TEXT NOT NULL after output;
      ALTER TABLE nagios_hostchecks ADD long_output TEXT NOT NULL after output;
      ALTER TABLE nagios_hoststatus ADD long_output TEXT NOT NULL after output;
      ALTER TABLE nagios_notifications ADD long_output TEXT NOT NULL after output;
      ALTER TABLE nagios_servicechecks ADD long_output TEXT NOT NULL after output;
      ALTER TABLE nagios_servicestatus ADD long_output TEXT NOT NULL after output;
      ALTER TABLE nagios_statehistory ADD long_output TEXT NOT NULL after output;
      ALTER TABLE nagios_systemcommands ADD long_output TEXT NOT NULL after output;

      삭제
  19. Hi Yeonki,
    Thank you for this post, It makes my job easy, Thanks a Lot.
    Keep up your good work.

    답글삭제
  20. 작성자가 댓글을 삭제했습니다.

    답글삭제
  21. 오 nagios세팅하다 찾은글이! 많은 도움됐습니다! ㅎㅎㅎ

    답글삭제
    답글
    1. 오랜만이에요.. 도움이 됐다니 다행이네요. ㅋㅋ

      삭제
  22. Hi,

    Thanks for giving such helpful guide.

    But I am facing one problem as below:

    Everything is working well but there is no data written in tables.

    Please check my log as below:

    [1400039127] ndomod: NDOMOD 1.5.2 (06-08-2012) Copyright (c) 2009 Nagios Core Development Team and Community Contributors
    [1400039127] ndomod: Successfully connected to data sink. 0 queued items to flush.
    [1400039127] Event broker module '/usr/local/ndoutils-1.5.2/bin/ndomod.o' initialized successfully.
    [1400039127] Finished daemonizing... (New PID=14890)

    Please help me because I am working on same from last two days. I have already reinstalled ndoutils because of this issue.

    답글삭제
  23. Hi.
    It seemed your environment was normal based on log entries.
    It is hard to decide what is wrong at the moment.
    Could you post steps what you installed ndoutil and your OS info ?

    답글삭제
  24. Hi,

    Thanks for Reply.

    I have followed steps which are described in this blog.

    And my OS info is Ubuntu 12.10.

    Is there any server issue?

    답글삭제
    답글
    1. Hi, Jay,
      I am not sure.
      I need to check if I have the same problem like you when I install ndoutil on Ubuntu 12.10.
      I'll leave comments after that.

      삭제
    2. Hi, Jay.
      My how-to was not working on Ubuntu 12.10.
      I downloaded Nagios 4.0.6 and Ndotutils 2.0 and they weren't put data into MySQL. I need further investigating in this regard.

      However, Ubuntu provides nagios packages. Nagios 3.4 and Ndoutils 1.4b9. I made sure they were working on Ubuntu 12.10
      If you'd like to install these, just type the following:
      $ sudo apt-get install nagios3 nagios3-cgi nagios-plugins ndoutils-nagios3-mysql

      FYI, I also tested these on CentOS 6.5. this time, they were suucceeded to put data into DB.

      I hope this will help you.

      삭제
  25. Hi,

    I have tried to install plugin by using given command but though it was not put data in Database.

    답글삭제
  26. Hi,

    Can you please tell me where can I find all settings and database detail(like ndo2db.cfg/ndo2mod.cfg and event_broker module) for this type of ndoutils installation?

    And also I am not getting any log regarding to ndoutils in Nagios log so I think that there is an issue with linking ndoutils with nagios3.

    답글삭제
    답글
    1. Hi,

      As to my whole configuration files of nagios3 ubuntu packages, please refer to here: http://paste.openstack.org/show/80971/

      This containtains:
      1) contents of each configuration file (nagios.cfg, ndo2db.cfg and ndomod.cfg)
      2) To start ndoutils as a standalone, need to changed in /etc/default/ndoutils, After changed, should restart it (sudo service ndoutils restart)

      삭제
  27. Hi,

    I forgot to add event_broker in nagios.cfg for standalone version.

    Now everything is working perfectly.

    Thanks..:)

    답글삭제
  28. Hi..I am unable to start the ndo2db service
    Error:Starting ndo2db I am getting the error: "Support for the specified database server is either not yet supported, or was not found on your system."
    I have installed the MySQL Community Server 5.5.38 Protocol version 10.
    Database is also available in the DB.Can you please Please help me because I am working on same from last three days.Reinstalled too.

    답글삭제