Friday, April 22, 2011

Sys Admin Tools 0.3 -- Recap/Documentation

Welcome to my 0.3 release of my Sys Admin Tools -- Implementing Icinga Monitoring System. This post will deliver a PDF file providing a recap on my previous posts.

This PDF file can be found here -- note that this file is a generic documentation, for more detailed instructions please follow my previous posts.

To easily access my previous blog posts which contain much more detailed information regarding my implementation of Icinga I have created several PDF files of my previous releases, you may view these releases individually by clicking on the appropriate one below;

0.1 Release & 0.1 Release Continue
0.2 Release
0.3 Release

Stay tuned for my next release which will contain upgrades and a much more detailed PDF documentation!

Monday, April 18, 2011

Creating a YUM Repository

Welcome Again. In this post I will describe the steps required in order to create your own YUM repository that can be used to install new versions of software. For this post I am using the “gnupg-1.4.11-1.fc14.x86_64.rpm” I created in my previous posts. Only this time I had added an option in the Spec File which increased the Epoch version.

Before we begin you need to generate a GPG Key using the gpg --gen-key command -- this allows you to sign your RPM file.

After you generate your key you will need to add the email address associated to your GPG key to the %_gpg_name macro in the ~/.rpmmacros file -- %_gpg_name “email-address”.

Our first step after creating our GPG key would be to sign this RPM package, the command that we need to use in order to accomplish this is below – enter your pass phrase when prompted;
rpm --addsign gnupg-1.4.11-1.fc14.x86_64.rpm
Enter pass phrase:
Pass phrase is good.

If the previous command returned successful then we can move onto the next step – creating your repository. (Assuming Apache is installed we will create the directory /var/www/html/fedora/14/x86_64/rpms/ which will hold our RPM files). To accomplish this follow the steps provided below;

Log in as the super user
cp ~/rpmbuild/RPMS/gnupg-1.4.11-1.fc14.x86_64.rpm  /var/www/html/fedora/14/x86_64/rpms/
createrepo /var/www/html/fedora/14/x86_64/rpms/ (output provided below)
1/1 - gnupg-1.4.11-1.fc14.x86_64.rpm                                           
Saving Primary metadata
Saving file lists metadata
Saving other metadata

If the YUM repository was created successfully above you should see a directory named repodata in the /var/www/html/fedora/14/x86_64/rpms/ directory.

After we have verified that the repository has been created you will need to create a new repository file inside of the /etc/yum.repos.d directory -- an example of a repository file (called fedora-gcasella.repo) I have created is below;
[fedora-gcasella]
name=Fedora $releasever - $basearch
failovermethod=priority
baseurl=http://gcasella.dynamic-dns.net/fedora/14/$basearch/rpms/
enabled=1
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-gcasella

IMPORTANT: If the gpgkey option is commented out and you try to run the yum install command you will receive an error indicating that the public key for the package is not installed.

Before testing out new repository we will create a new file within the /etc/pki/rpm-gpg/ directory – you may give it any name that you want.

After creating this file we will run the command; gpg --export --armor <email-address> (ensure that this is done as the regular user and NOT the super user.) Save the output of this command to the new file you had created inside of the /etc/pki/rpm-gpg/ directory.
If everything has gone well you will be able to test your newly created repository (ensure that the httpd service is started on the machine holding your RPM files.) and run the yum install gnupg command. You should receive similar output as below;

Loaded plugins: langpacks, presto, refresh-packagekit
Adding en_US to language list
fedora-gcasella                                                                                                                       | 1.3 kB     00:00    
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package gnupg.x86_64 3:1.4.11-1.fc14 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================================
 Package                         Arch                             Version                                    Repository                                 Size
============================================================================================================================================================
Updating:
 gnupg                           x86_64                           3:1.4.11-1.fc14                            fedora-gcasella                           1.2 M

Transaction Summary
============================================================================================================================================================
Upgrade       1 Package(s)

Total size: 1.2 M
Is this ok [y/N]: y
Downloading Packages:
warning: rpmts_HdrFromFdno: Header V4 RSA/SHA1 Signature, key ID e1af8018: NOKEY
fedora-gcasella/gpgkey                                                                                                                | 3.4 kB     00:00 ...
Importing GPG key 0xE1AF8018:
Userid: "Gian-Luca Casella <gcasella@learn.senecac.on.ca>"
From  : /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-gcasella
Is this ok [y/N]: y

Selecting “y” will install the GPG key into your system, indicating that it is now a valid signature and to accept any other packages from you that you may have available for download.


Repository-Release RPM:

To make it easier for other users to access my repository I have created an RPM which contains both my repository file along with my GPG key. This RPM can be found by clicking on this link.

The steps below show how I was able to create this repository release RPM file …
cp /etc/yum.repos.d/fedora-gcasella.repo   ~/rpmbuild/SOURCES/
cp /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-gcasella ~/rpmbuild/SOURCES/
tar cvf gcasellarepo-0.0.1.tar *gcasella* && gzip gcasellarepo-0.0.1.tar

The commands above will copy the repository file and the GPG Key file (created previously) into your ~/rpmbuild/SOURCES/ directory. Next you would need to create a tarball of these files using the tar cvf command – this tarball can be found here.

Once we have the Sources in the appropriate place we will create the SPEC file, the contents of my spec file are listed below, this will install the repository file and the GPG key to their appropriate locations;

Name:gcasellarepo
Version:0.0.1
Release:1%{?dist}
Summary:RPM Package containing my repository file along with my GPG Key

Group:System Environment/Base
License:GPLv3
URL:http://gcasella.dynamic-dns.net/fedora/
Source0:http://gcasella.dynamic-dns.net/fedora/14/x86_64/source/gcasellarepo-0.0.1.tar.gz
BuildRoot:%{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)

%description
Installs my custom made repository configuration file along with its GPG key

%prep
%setup -q

%build

%install
rm -rf $RPM_BUILD_ROOT
mkdir -p $RPM_BUILD_ROOT%{_sysconfdir}/yum.repos.d/
mkdir -p $RPM_BUILD_ROOT%{_sysconfdir}/pki/rpm-gpg/
cp -p fedora-gcasella.repo $RPM_BUILD_ROOT%{_sysconfdir}/yum.repos.d/
cp -p RPM-GPG-KEY-fedora-gcasella $RPM_BUILD_ROOT%{_sysconfdir}/pki/rpm-gpg/

%clean
rm -rf $RPM_BUILD_ROOT

%files
%defattr(-,root,root,-)
%doc
/etc/yum.repos.d/fedora-gcasella.repo
/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-gcasella

%changelog
* Tue Feb 08 2011 Gian-Luca Casella <gcasella@learn.senecac.on.ca> 0.0.1
- Created my own personal RPM in order to install both my repository file along with the corresponding GPG Key needed to install with.

Additional Notes:

If you buy chance have SELinux running and is set to “Enforcing” issue the setsebool  -P httpd_read_user_content 1 command, this has fixed any issues for me.

Gian-Luca Casella -- Last Updated on Monday, April 18, 2011

Monday, April 4, 2011

Sys Admin Tools 0.2 -- Implementation

Welcome to my 0.2 release of Icinga. In this release I will discuss how we can install Icinga from an RPM file and modify its configuration files in order to allow us to monitor Fedora ARM machines. This RPM was compiled for Fedora 14 from a colleague of mine Mike Kirton, but because I am running Icinga on a FC12 machine I simply built the package again on my Fedora 12 test machine using the rpmbuild -ba command.

Before we continue I want to note the types of things that Icinga will allow us to monitor.

Private Services
   CPU Load
   Memory Usage
   Disk Usage
   Logged In Users
   Running Processes
   And much more…

Public Services
   HTTP
   FTP
   SSH
   SMTP
   And much more…

Icinga Installation
There are a few necessary packages that we will need to install before we begin with the installation of Icinga. Issue the following command to obtain them;
yum install nagios-plugins nagios-plugins-all nagios-plugins-nrpe nrpe

Next we will need to obtain the necessary RPM files that were created while packaging Icinga for Fedora 12, using the simple command will obtain all of the necessary packages;
cd ~; wget -r -l1 --no-parent -A.rpm http://gcasella.dynamic-dns.net/sbr/icinga/fc12/

This will download the following packages inside of the ~/gcasella.dynamic-dns.net/sbr/icinga/fc12/ directory;
icinga-1.3.0-1.fc12.x86_64.rpm
icinga-api-1.3.0-1.fc12.x86_64.rpm
icinga-debuginfo-1.3.0-1.fc12.x86_64.rpm
icinga-doc-1.3.0-1.fc12.x86_64.rpm
icinga-gui-1.3.0-1.fc12.x86_64.rpm
icinga-idoutils-1.3.0-1.fc12.x86_64.rpm

Note that the order of installation is quite important with these RPM files, use this command below to install them in their appropriate order (copy & paste it);
rpm -i icinga-doc-1.3.0-1.fc12.x86_64.rpm  &&  rpm ii icinga-1.3.0-1.fc12.x86_64  &&  rpm -i icinga-api-1.3.0-1.fc12.x86_64.rpm  &&  rpm –i icinga-gui-1.3.0-1.fc12.x86_64.rpm

If you receive no errors after installation then we can continue on to configuration, if you do a simple query of Icinga you will see what files/directories were created/installed, we will be working with most of these files (rpm -ql icinga).

Configuring Icinga
Congratulations, you have successfully installed Icinga, now what we will want to do is make all the necessary configuration changes before we attempt to start the Icinga service.

To make the configuration process easier to read we will want to create these three files in the /etc/icinga/ directory, they are created to organize our remote hosts making it much cleaner to read. The command below will create the four extra files.
cd /etc/icinga; touch remote-linux-hosts.cfg hostgroups.cfg services.cfg remote-linux-template.cfg

  • /etc/icinga/objects/remote-linux-hosts.cfg -- will contain all of the hosts that you want to monitor.
  • /etc/icinga/objects/hostgroups.cfg – will allow you to map a simple hostgroup_name to a series of remote machines.
  • /etc/icinga/objects/sevices.cfg – will contain all of the services you would like to be monitored on your remote machines – this is when you can use the hostgroup_name.
  • /etc/icinga/objects/remote-linux-template.cfg – a simple template that will be used for all of the hosts that we are defining, for example their notification period, when it should send notifications, etc. – this will be discussed later in this release.

Next we are going to take a look at the changes we will need to make to all of necessary files in order for Icinga to start properly. A list of the files that must be modified are listed below -- in addition the contents of the files and what will need to be modified in order for us to start Icinga;

/etc/icinga/icinga.cfg
This is the main configuration file for Icinga. If new files are added to the /etc/icinga/objects/ directory and are not added in this configuration file, then you will receive an error when trying to start the service. Because of the new files we created earlier we will need to modify this file and make the following additions anywhere in the file;
cfg_file=/etc/icinga/objects/remote-linux-hosts.cfg
cfg_file=/etc/icinga/objects/hostgroups.cfg
cfg_file=/etc/icinga/objects/services.cfg
cfg_file=/etc/icinga/objects/remote-linux-template.cfg

/etc/icinga/objects/contacts.cfg
This configuration file allows us to set the email address that will be used to send notifications to. This file will also allow us to add more members to the admins contactgroup – who will also receive emails. This file does not have to be modified greatly; you will just need to put your email address in replace of the default as shown below – which is shown by the red font;
define contact  {
        contact_name                 icingaadmin      
        use                                   generic-contact
        alias                                 Icinga Admin
        email                                icingaadmin@hostname         
}

/etc/icinga/objects/remote-linux-template.cfg
This configuration file will hold the template for all of the hosts that we will monitor remotely, Icinga already had premade templates, but I feel it makes things much more flexible when you create your own, for a description of what each option in the template does you can refer to the Nagios Object Definitions;
define host {
        name                                           remote-fedora-arm-template
        use                                              generic-host     
        notifications_enabled                  1        
        event_handler_enabled               1        
        flap_detection_enabled               1        
        failure_prediction_enabled          1        
        process_perf_data                      1        
        retain_status_information           1        
        retain_nonstatus_information    1        
        notification_period                       24x7    
        check_period                              24x7   
        check_interval                             2     
        retry_interval                               1    
        max_check_attempts                 10 
        check_command                                    check-host-alive
        notification_period                       24x7
        notification_interva                      30      
        notification_options                     d,u,r  
        contact_groups                           admins
        register                                        0
}

/etc/icinga/objects/remote-linux-hosts.cfg
This configuration file is the place holder for all of the hosts that you want to monitor. The hosts that are present in this configuration file will be mapped later on inside of the /etc/icinga/objects/hostgroups.cfg file. I will provide two hosts in this file for the purpose of this release;

define host {
        use                      remote-fedora-arm-template  #from our template config file
        host_name          CN-ICINGAH.casellanetworks.ca
        alias                     CN-ICINGAH
        address               67.35.20.113
        register                1
}

define host {
        use                      remote-fedora-arm-template #from our template config file
        host_name          CN-FUNC.casellanetworks.ca
        alias                     CN-FUNC
        address               67.35.20.109
        register                1
}

IMPORTANT NOTE: If you would like to add more hosts to this file just simply copy and paste any host in this file and make the proper changes to their hostname and IP Address.

/etc/icinga/objects/hostgroups.cfg
This configuration file will allow us to map multiple hosts to a single name which will be later used in the /etc/icinga/objects/services.cfg file. Using the hosts above I will provide an example of this configuration file;

define hostgroup {
        hostgroup_name    remote-fc-arm ;used in services.cfg
        alias                         Fedora ARM
        members                CN-ICINGAH.casellanetworks.ca,CN-FUNC.casellanetworks.ca        
}

/etc/icinga/objects/services.cfg
This configuration file will be the placeholder of all the services you would like Icinga to monitor on your remote machines, as noted at the beginning of this post you have a number of public and private services that can be monitored. This is the file in which they will go, using the hostgroup_name above to determine which hosts it will actually monitor – I will only show two services for the purpose of this release, Icinga by default installs with a series of services already listed in their configuration files. You may refer to the Nagios Documentation for Monitoring Public Services for additional information;

define service {
        use                                 generic-service         ; Name of service template to use
        hostgroup_name          remote-fc-arm             ; Created in hostgroups.cfg
        service_description     PING
        check_command         check_ping!100.0,20%!500.0,60%
        }

define service {
        use                                  generic-service         ; Name of service template to use
        hostgroup_name             remote-fc-arm
        service_description         Root Partition
        check_command                        check_local_disk!20%!10%!/
        }


Setting up Remote Systems
This is a very simple process. For us to make sure that our remote systems communicate properly with our Icinga server we need to install the Nagios plugins and the NRPE package and make slight modifications to its configuration.

To begin you will need to log into the remote system that you would like to monitor and issue the following command; yum install nagios-plugins-all nagios-plugins-nrpe nrpe

The files that we need to edit are listed below along with the modifications that will need to be made to them;

/etc/nagios/nrpe.cfg
This file will allow us to change the port that NRPE will use to communicate with our Icinga server, but the only portion of this file that will need to be changed is the allowed_hosts option. Multiple hosts can be separated by a comma. This option will tell NRPE the IP Address of the Icinga server allowing only that server to connect into the system.

/etc/sysconfig/nrpe
This file is simple, it gives us the option to make NRPE run in SSL mode, according to Icinga and Nagios documentation it is best practice to keep the communication between your Icinga server and remote host encrypted. This can be done by modifying the one and only line in this file by adding -n to the NRPE_SSL_OPT option.

Next we will make sure that our NRPE service starts on boot by using the chkconfig nrpe on command and then finally start our service using the /etc/init.d/nrpe start command.

In order for us to test that NRPE is functional we will issue this very simple command;
/usr/lib64/nagios/plugins/check_nrpe -H 127.0.0.1 -n
(The -n argument makes the check_nrpe command use SSL to connect)

If it is successful you should see a response of “NRPE v2.12”.

IMPORTANT: If iptables is enabled you will need to add the rule that will allow port 5666, this can be done using the command; iptables -I INPUT -p tcp -m tcp --dport 5666 -j ACCEPT

If you would like to read any additional notes on NRPE and the setup of it you can refer to the Icinga NRPE documents.


Conclusion
If all is well you may attempt to start you Icinga service by running the /etc/init.d/icinga start command, if something has gone wrong then it will tell you check its log. If the starting of the service is successful we will then need to reload the Apache webserver with the /etc/init.d/httpd reload command – this will only re-read the configuration files and not restart the actual service. If no other errors persist visit your Icinga Classic Web Interface (http://host-name.com/icinga/) and log in with the credentials you created above with the htpasswd command; username: icingaadmin  password: <your-password>.

To ensure that everything is running smoothly I will perform a quick recap of what services we need running, and what should be running on startup;
-       Apache (httpd)
-       Sendmail
-       Icinga
-       NRPE (Only running on the remote hosts)


This concludes my 0.2 release, please stay tuned for my final 0.3 release.



Gian-Luca Casella -- Last Updated on Monday, April 4, 2011

Saturday, March 26, 2011

Sys Admin Tools 0.1 - Installing and Monitoring with Icinga

Welcome to the third and final part of my 0.1 release. In this post I will discuss the steps necessary in order to in install and configure Icinga from source – at a later time I will post about installation from an RPM which a colleague of mine is working on.


We must install the prerequisites for Icinga before we move on, these include; Apache, GCC Compiler, C/C++ development libraries, and GD development libraries. Use the following command (as root or sudo) in order to install the necessary packages for RedHat Based distributions;

            yum install httpd gcc glibc glibc-common gd gd-devel libjpeg libjpeg-devel
            libpng libpng-devel net-snmp net-snmp-devel

Once you have the prerequisites installed you will need to the follow the upcoming steps in order to successfully install your Icinga Core and Icinga Web from source code.

Create your new icinga user account and give it a password
useradd –m icinga && passwd icinga

Create the required group named “icinga”
groupadd  icinga

We will then need to create a new group called “icinga-cmd” and adding the systems web user and Icinga user to this group will allow for them to use the classic web interface to send commands to Icinga.
groupadd icinga-cmd
usermod –a –G icinga-cmd icinga
usermod –a –G icinga-cmd apache

Change your current working directory to a location of your choosing and download and extract the following tarball files (as root or sudo) – the purpose of this post we will use “/usr/src” as our directory;
cd /usr/src/

Files to download;

wget http://sourceforge.net/projects/icinga/files/icinga/1.3.0/icinga-1.3.0.tar.gz/download
wget http://sourceforge.net/projects/icinga/files/icinga-web/1.3.0/icinga-web-1.3.0.tar.gz/download
wget http://sourceforge.net/projects/nagiosplug/files/nagiosplug/1.4.15/nagios-plugins-1.4.15.tar.gz/download

Next we will extract the Icinga Core and prepare it for installation;

tar –zxvf  icinga-1.3.0.tar.gz
cd /usr/src/icinga-1.3.0/
./configure --with-command-group=icinga-cmd (use the --help arguments for additional options)
            make all

To complete a full installation of Icinga from source you can simply enter the following command; “make fullinstall” – the default installation directory is “/usr/local/icinga/”. (Note: Do not start Icinga yet!)

Before we continue we must change the email address inside of the “/usr/local/icinga/etc/objects/contacts.cfg file – this address is used for notification emails, review the other options in the file for additional changes you may want to make.

Icinga does come with a Classic Web Interface already built in, use the following commands to set up and configure your web interface that will be used to administer your Icinga server – if you would like to know how to install the new Icinga-Web follow this link.
cd /usr/src/icinga-1.3.0/
            make cgis
            make install-cgis
            make install-html
            make install-webconf

Create the icingaadmin account that will be logging into the Icinga classic web interface and then restart your web server – the account is determined by the entry found in the “/usr/loca/icinga/etc/objects/contacts.cfg” file’s “contact_name” option.

            htpasswd -c /usr/local/icinga/etc/htpasswd.users icingaadmin
            /etc/init.d/httpd restart

Because Icinga is backwards compitable with Nagios, it uses all of Nagios’ available plugins – these will need to be installed as well.

            cd /usr/src/nagios-plugins-1.4.15
            ./configure --prefix=/usr/loca/icinga --with-cgiurl=/icinga/cgi-bin --with-htmurl=/icinga --with-nagios-user=icinga --with-nagios-group=icinga
            make && make install


IMPORTANT: The commands above much also be executed on ALL machines that are to be monitored by Icinga. 

You may run into the always annoying encounters with SELinux settings on RedHat distributions, to perform an easy fix you can simply set your SELinux settings to permissive mode (setenforce 0) – the Icinga documentation provides the necessary commands in order to fix the SELinux issue, the commands are found below;
            
            chcon –R –t httpd_sys_script_exec_t /usr/local/icinga/sbin/
            chcon –R –t httpd_sys_content_t /usr/local/icinga/share/
            chcon –t httpd_sys_script_rw_t /usr/local/icinga/var/rw/icinga.cmd

This concludes the installation of your new Icinga server, you are now able to start your Icinga service – be sure to add it to your start up scripts;
            chkconfig --add icinga
            chkconfig icinga on
            /etc/init.d/icinga start

In order to access your Classic Web Interface you will need to redirect yourself to the following URL – http://localhost/icinga/ or http://yourdomain.com/icinga/
IMPORTANT: If iptables is enabled you MUST add the exception for port 80 to be accepted –     iptables –A INPUT –p tcp –m tcp –dport 80 –j ACCEPT


The following video will show a very small demonstration of what Icinga looks like after it is installed, with also a simple notification sample. Note that it is a demonstration on my virtual servers at home and is not yet ready for production. 



To be continued... Stay tuned for my 0.2 release where I will 
demonstrate Icinga in action, showing configuration 
and real-time monitoring on Fedora-ARM machines.



Gian-Luca Casella -- Last Updated on Tuesday, March 26, 2011