Monday, April 4, 2011

Sys Admin Tools 0.2 -- Implementation

Welcome to my 0.2 release of Icinga. In this release I will discuss how we can install Icinga from an RPM file and modify its configuration files in order to allow us to monitor Fedora ARM machines. This RPM was compiled for Fedora 14 from a colleague of mine Mike Kirton, but because I am running Icinga on a FC12 machine I simply built the package again on my Fedora 12 test machine using the rpmbuild -ba command.

Before we continue I want to note the types of things that Icinga will allow us to monitor.

Private Services
   CPU Load
   Memory Usage
   Disk Usage
   Logged In Users
   Running Processes
   And much more…

Public Services
   HTTP
   FTP
   SSH
   SMTP
   And much more…

Icinga Installation
There are a few necessary packages that we will need to install before we begin with the installation of Icinga. Issue the following command to obtain them;
yum install nagios-plugins nagios-plugins-all nagios-plugins-nrpe nrpe

Next we will need to obtain the necessary RPM files that were created while packaging Icinga for Fedora 12, using the simple command will obtain all of the necessary packages;
cd ~; wget -r -l1 --no-parent -A.rpm http://gcasella.dynamic-dns.net/sbr/icinga/fc12/

This will download the following packages inside of the ~/gcasella.dynamic-dns.net/sbr/icinga/fc12/ directory;
icinga-1.3.0-1.fc12.x86_64.rpm
icinga-api-1.3.0-1.fc12.x86_64.rpm
icinga-debuginfo-1.3.0-1.fc12.x86_64.rpm
icinga-doc-1.3.0-1.fc12.x86_64.rpm
icinga-gui-1.3.0-1.fc12.x86_64.rpm
icinga-idoutils-1.3.0-1.fc12.x86_64.rpm

Note that the order of installation is quite important with these RPM files, use this command below to install them in their appropriate order (copy & paste it);
rpm -i icinga-doc-1.3.0-1.fc12.x86_64.rpm  &&  rpm ii icinga-1.3.0-1.fc12.x86_64  &&  rpm -i icinga-api-1.3.0-1.fc12.x86_64.rpm  &&  rpm –i icinga-gui-1.3.0-1.fc12.x86_64.rpm

If you receive no errors after installation then we can continue on to configuration, if you do a simple query of Icinga you will see what files/directories were created/installed, we will be working with most of these files (rpm -ql icinga).

Configuring Icinga
Congratulations, you have successfully installed Icinga, now what we will want to do is make all the necessary configuration changes before we attempt to start the Icinga service.

To make the configuration process easier to read we will want to create these three files in the /etc/icinga/ directory, they are created to organize our remote hosts making it much cleaner to read. The command below will create the four extra files.
cd /etc/icinga; touch remote-linux-hosts.cfg hostgroups.cfg services.cfg remote-linux-template.cfg

  • /etc/icinga/objects/remote-linux-hosts.cfg -- will contain all of the hosts that you want to monitor.
  • /etc/icinga/objects/hostgroups.cfg – will allow you to map a simple hostgroup_name to a series of remote machines.
  • /etc/icinga/objects/sevices.cfg – will contain all of the services you would like to be monitored on your remote machines – this is when you can use the hostgroup_name.
  • /etc/icinga/objects/remote-linux-template.cfg – a simple template that will be used for all of the hosts that we are defining, for example their notification period, when it should send notifications, etc. – this will be discussed later in this release.

Next we are going to take a look at the changes we will need to make to all of necessary files in order for Icinga to start properly. A list of the files that must be modified are listed below -- in addition the contents of the files and what will need to be modified in order for us to start Icinga;

/etc/icinga/icinga.cfg
This is the main configuration file for Icinga. If new files are added to the /etc/icinga/objects/ directory and are not added in this configuration file, then you will receive an error when trying to start the service. Because of the new files we created earlier we will need to modify this file and make the following additions anywhere in the file;
cfg_file=/etc/icinga/objects/remote-linux-hosts.cfg
cfg_file=/etc/icinga/objects/hostgroups.cfg
cfg_file=/etc/icinga/objects/services.cfg
cfg_file=/etc/icinga/objects/remote-linux-template.cfg

/etc/icinga/objects/contacts.cfg
This configuration file allows us to set the email address that will be used to send notifications to. This file will also allow us to add more members to the admins contactgroup – who will also receive emails. This file does not have to be modified greatly; you will just need to put your email address in replace of the default as shown below – which is shown by the red font;
define contact  {
        contact_name                 icingaadmin      
        use                                   generic-contact
        alias                                 Icinga Admin
        email                                icingaadmin@hostname         
}

/etc/icinga/objects/remote-linux-template.cfg
This configuration file will hold the template for all of the hosts that we will monitor remotely, Icinga already had premade templates, but I feel it makes things much more flexible when you create your own, for a description of what each option in the template does you can refer to the Nagios Object Definitions;
define host {
        name                                           remote-fedora-arm-template
        use                                              generic-host     
        notifications_enabled                  1        
        event_handler_enabled               1        
        flap_detection_enabled               1        
        failure_prediction_enabled          1        
        process_perf_data                      1        
        retain_status_information           1        
        retain_nonstatus_information    1        
        notification_period                       24x7    
        check_period                              24x7   
        check_interval                             2     
        retry_interval                               1    
        max_check_attempts                 10 
        check_command                                    check-host-alive
        notification_period                       24x7
        notification_interva                      30      
        notification_options                     d,u,r  
        contact_groups                           admins
        register                                        0
}

/etc/icinga/objects/remote-linux-hosts.cfg
This configuration file is the place holder for all of the hosts that you want to monitor. The hosts that are present in this configuration file will be mapped later on inside of the /etc/icinga/objects/hostgroups.cfg file. I will provide two hosts in this file for the purpose of this release;

define host {
        use                      remote-fedora-arm-template  #from our template config file
        host_name          CN-ICINGAH.casellanetworks.ca
        alias                     CN-ICINGAH
        address               67.35.20.113
        register                1
}

define host {
        use                      remote-fedora-arm-template #from our template config file
        host_name          CN-FUNC.casellanetworks.ca
        alias                     CN-FUNC
        address               67.35.20.109
        register                1
}

IMPORTANT NOTE: If you would like to add more hosts to this file just simply copy and paste any host in this file and make the proper changes to their hostname and IP Address.

/etc/icinga/objects/hostgroups.cfg
This configuration file will allow us to map multiple hosts to a single name which will be later used in the /etc/icinga/objects/services.cfg file. Using the hosts above I will provide an example of this configuration file;

define hostgroup {
        hostgroup_name    remote-fc-arm ;used in services.cfg
        alias                         Fedora ARM
        members                CN-ICINGAH.casellanetworks.ca,CN-FUNC.casellanetworks.ca        
}

/etc/icinga/objects/services.cfg
This configuration file will be the placeholder of all the services you would like Icinga to monitor on your remote machines, as noted at the beginning of this post you have a number of public and private services that can be monitored. This is the file in which they will go, using the hostgroup_name above to determine which hosts it will actually monitor – I will only show two services for the purpose of this release, Icinga by default installs with a series of services already listed in their configuration files. You may refer to the Nagios Documentation for Monitoring Public Services for additional information;

define service {
        use                                 generic-service         ; Name of service template to use
        hostgroup_name          remote-fc-arm             ; Created in hostgroups.cfg
        service_description     PING
        check_command         check_ping!100.0,20%!500.0,60%
        }

define service {
        use                                  generic-service         ; Name of service template to use
        hostgroup_name             remote-fc-arm
        service_description         Root Partition
        check_command                        check_local_disk!20%!10%!/
        }


Setting up Remote Systems
This is a very simple process. For us to make sure that our remote systems communicate properly with our Icinga server we need to install the Nagios plugins and the NRPE package and make slight modifications to its configuration.

To begin you will need to log into the remote system that you would like to monitor and issue the following command; yum install nagios-plugins-all nagios-plugins-nrpe nrpe

The files that we need to edit are listed below along with the modifications that will need to be made to them;

/etc/nagios/nrpe.cfg
This file will allow us to change the port that NRPE will use to communicate with our Icinga server, but the only portion of this file that will need to be changed is the allowed_hosts option. Multiple hosts can be separated by a comma. This option will tell NRPE the IP Address of the Icinga server allowing only that server to connect into the system.

/etc/sysconfig/nrpe
This file is simple, it gives us the option to make NRPE run in SSL mode, according to Icinga and Nagios documentation it is best practice to keep the communication between your Icinga server and remote host encrypted. This can be done by modifying the one and only line in this file by adding -n to the NRPE_SSL_OPT option.

Next we will make sure that our NRPE service starts on boot by using the chkconfig nrpe on command and then finally start our service using the /etc/init.d/nrpe start command.

In order for us to test that NRPE is functional we will issue this very simple command;
/usr/lib64/nagios/plugins/check_nrpe -H 127.0.0.1 -n
(The -n argument makes the check_nrpe command use SSL to connect)

If it is successful you should see a response of “NRPE v2.12”.

IMPORTANT: If iptables is enabled you will need to add the rule that will allow port 5666, this can be done using the command; iptables -I INPUT -p tcp -m tcp --dport 5666 -j ACCEPT

If you would like to read any additional notes on NRPE and the setup of it you can refer to the Icinga NRPE documents.


Conclusion
If all is well you may attempt to start you Icinga service by running the /etc/init.d/icinga start command, if something has gone wrong then it will tell you check its log. If the starting of the service is successful we will then need to reload the Apache webserver with the /etc/init.d/httpd reload command – this will only re-read the configuration files and not restart the actual service. If no other errors persist visit your Icinga Classic Web Interface (http://host-name.com/icinga/) and log in with the credentials you created above with the htpasswd command; username: icingaadmin  password: <your-password>.

To ensure that everything is running smoothly I will perform a quick recap of what services we need running, and what should be running on startup;
-       Apache (httpd)
-       Sendmail
-       Icinga
-       NRPE (Only running on the remote hosts)


This concludes my 0.2 release, please stay tuned for my final 0.3 release.



Gian-Luca Casella -- Last Updated on Monday, April 4, 2011

No comments:

Post a Comment