You are asked to escalate a down host or service to either another technical level or add an incident to your Change Management System. What do you do? Nagios has a great object attribute called escalations for you to configure for the purpose of escalating host and or service issues.

If you already have an existing Nagios host and service monitoring and notification system established, you will be up and running in two steps:

1. Add any new contacts and or contact groups.
2. Add the escalation configuration.

I will explain how I added an escalation to notify a Change Management System on the first notification. Once the Change Request is sent you will no longer have to create any more problem tickets, since the technician working the problem should do two things: Acknowledge the problem through Nagios and work the problem ticket.

1. Add any new contacts for the escalation. I had to add a contact for the Change Management System, since we are able to open an incident using email.

name-of-your-contact-file.cfg

define contactgroup{
	contactgroup_name		name of your cms
	alias				Name of Your CMS
	members				name_of_your_cms
	}
define contact{
        contact_name                    name_of_your_cms
        alias                           Name_of_Your_CMS
	contact_groups			name_of_your_cms
        host_notifications_enabled      1
        service_notifications_enabled   1
        service_notification_period     24x7
        host_notification_period        24x7
        service_notification_options    c #We only need a problem ticket open when the service is critical
        host_notification_options       d #We only need a problem ticket open when the host is down
        service_notification_commands   notify-linux-service-by-email
        host_notification_commands      notify-linux-host-by-email
       email                           email_address@your_domain.com
        can_submit_commands             1
        }

2. Add the escalation.

Name_of_your_escalation.cfg

 define hostescalation{
 	hostgroup_name		name_of_your_hostgroup
 	first_notification	1
 	last_notification	1
 	notification_interval	5
 	contact_groups		name_of_your_cms_group
	escalation_period	timeperiod_to_notify_create_incident #ex. 24x7
	escalation_options      d #We want a problem ticket created when the host is down.
 	}

Now, you need to remember how a service escalation works. The escalation is read into Nagios during the reload. When the escalation definition completes, Nagios is smart enough to start with the notification attributes defined in your host and or service template. So, in this instance Nagios will perform the escalation definition and notify your Change Management System once (1). Nagios will continue notifications based upon your notification definitions within the template.

Like everything in life there is a catch. If you create an escalation with a contact group defined within your template, Nagios will only execute the escalation file. For example, if you have a tech_email contact group in the host or service definition and you add it to the CMS contact group within the escalation template both groups will only be notified once (1) when a host or service is down or critical.

That is all there is to creating an escalation for a single purpose. Now, reload Nagios and the escalation will take effect. For service escalations you will be adding a service_definition. Play around with your escalations until you have the correct combination of attributes.

I recommend you add your email address to both groups and reload Nagios. Then select the Notification item in your web-based application. The notification section will show you the contact groups being notified. What I do is add a boggus host definition, add the host group to the escalations file, then reload Nagios.

Have fun and leave a comment if you have any questions or other suggestions for host and service escalations in Nagios.

Here is a group of files that you can put together in order to create a process to auto detect new hosts on your network when they are entered into DNS. If your DNS servers are Window Servers you will have to install the “dig” command and open the port to enable DNS searches.

Perform a “touch” command to create the new_hosts and found_this files.

README

Author: Mike Kniaziewicz
Purpose: Designed for Nagios 3.0 to dynamically search a domain and add newly found hosts to a host configuration file.
Email: mikhail@ebusinessjuncture.com
License: GNU Software License
Notice: There are no implicit or explicit warrantees associated with this software

Overview:
	auto_detect_hosts.pl was designed to gather information about the hosts in the DNS file for a particular domain. The software is meant to be run behind a firewall and is not for snooping on external domains. The software performs a 'dig' command, gathers the information, matches found host names against host names in Nagios, writes the found host names to new_hosts.cfg (all entries are commented out) and sends email to a specified email address with new hosts discovered.

Prerequisites:
	Must have Perl installed.
	Have all your host definitions in the same directory. This is so you are not searching unneccessary files.
	Modify the variables in auto_detect_hosts.pl file to match your enterprise requirements.

Installation:
	Place auto_detect_hosts,found_this and new_hosts in the same directory.
	Place new_hosts.cfg in the directory with your other host definitions.

	auto_detect_hosts.pl variables:

		my $email = 'userid\@domain.com'.
		my @mailer = [];		#array to hold new hosts for emailing
		my $domain_name = "com";	#Your domain name
		my $dig_cmd = "/usr/bin/dig";	#Path to "dig" command
		my @findings = []; 		#array to hold command returns
		my @lines = []; 		#array to hold search returns
		my $lines;			#holds individual lines
		my $path_to_hosts = "/$PREFIX/etc/objects/hosts/*"; #path to host config files
		my @ind_record; 		#array for individual findings
		my $counter = 0; 		#counter to cycle through array records
		NOTE: You will need to manually change some variables within the script to match your configurations. Examples are search string and file paths. I have changed the file paths to include $PREFIX so you can find where to make changes easily. If you are having problems running this script check first to make sure the path-to-perl is correct and the files have at least r+w permissions and the auto_detect_hosts.pl is executable.

	auto_detect_hosts.pl functions:

		sub mailer: Places newly found hosts and IP addresses into a file
		sub write_cfg_header: Writes the template information to the new_hosts.cfg file in the Nagios hosts directory.
		sub find_match: compairs the "dig" command results to hosts already in the Nagios hosts directory
		sub send_mail: Send email to the addresses listed in the email variable with a listing of new hosts found.

	Crontab Entry:
		In order to make this script fully functional you need to add it to your nagios crontab. Here is a sample entry to have the script run at 3:30 AM every day:
	30 3 * * * /$PREFIX/libexec/auto_detect_hosts.pl

After Thoughts:
	Any questions or suggestions please let me know at: mikhail@ebusinessjuncture.com.

auto_detect_new_hosts.pl

#!/usr/bin/perl -w

################################################################################

#Author:        Mikhail Kniaziewicz
#Date:          July 6, 2009
#Name:          auto_detect_hosts.pl
#Version:	1.0
#Purpose:       Auto detect host on a domain
#               using "dig"and add them to
#               the Nagios hosts file:
#               new_hosts.cfg with basic
#               PING command enabled
#
#Function Definitions
#sub mailer: Places newly found hosts and IP addresses into a file
#sub write_cfg_header: Writes the template information to the new_hosts.cfg filein the Nagios hosts directory.
#sub find_match: compairs the "dig" command results to hosts already in the Nagios hosts directory
#sub send_mail: Send email to the addresses listed in the email variable with a listing of new hosts found.

################################################################################

use strict;

################################################################################

#define variables
my $email = "userid\@domain.com"; #Where you want email sent
my @mailer = [];		#array to hold new hosts for emailing
my $domain_name = "domain.com";	#Your domain name
my $dig_cmd = "/usr/bin/dig";	#Path to "dig" command
my @findings = []; 		#array to hold command returns
my @lines = []; 		#array to hold search returns
my $lines;			#holds individual lines
my $path_to_hosts = "/$PREFIX/etc/objects/hosts/*"; #path to host config files
my @ind_record; 		#array for individual findings
my $counter = 0; 		#counter to cycle through array records

################################################################################
@findings = `$dig_cmd axfr $domain_name`; #put data into array
print `cp /dev/null /$PREFIX/libexec/new_hosts`;
open FH,">/$PREFIX/libexec/found_this" ||  die $!;
write_cfg_header(); #write the header is the file is empty
while ($findings[$counter]){
		if (($findings[$counter] =~ m/^\d+|\w+\.domain\.com/g) && ($findings[$counter] !~ /MX|SOA|CNAME|TXT|HINFO/)){
			$findings[$counter] =~ s/\t+|\s/:/g;
			chomp $findings[$counter];
			print  FH "$findings[$counter]\n";
		$counter++;
		}
		else {
		$counter++;
		}
	}
close(FH);
open FH,";
	 foreach $lines(@lines){
            chomp $lines;
            @ind_record = split(/:/,$lines,5);
	    $ind_record[0] =~ s/\.$//g;
	    $ind_record[4] =~ s/\:$//g;
		find_match($ind_record[0],$ind_record[4]);
         }
close(FH);
        my $filesize = -s  "/$PREFIX/libexec/new_hosts"; #file to check size
        if ($filesize != 0){
		send_mail(@mailer);
	}

sub write_cfg_header
{
	my $filesize = -s  "/$PREFIX/etc/objects/hosts/new_hosts.cfg"; #file to check size
	if ($filesize == 0){
	open FILE,">/$PREFIX/etc/objects/hosts/new_hosts.cfg" || die $!;
#Print the template information into the new_hosts.cfg file
	print FILE "# Generic host definition template - This is NOT a real host, just a template!
define host{
        name                            new_found_hosts    ; The name of this host template
        notifications_enabled           1       ; Host notifications are enabled
        event_handler_enabled           1       ; Host event handler is enabled
        flap_detection_enabled          1       ; Flap detection is enabled
        failure_prediction_enabled      1       ; Failure prediction is enabled
        process_perf_data               1       ; Process performance data
        retain_status_information       1       ; Retain status information across program restarts
        retain_nonstatus_information    1       ; Retain non-status information across program restarts
	check_period			workhours
	check_interval			3
	retry_interval                  1
	max_check_attempts              2
        active_checks_enabled           1
        check_command                   check-host-alive
	notification_period             workhours
	notification_interval           15
	notification_options            d,r
	contact_groups                  admins
        register                        0       ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL HOST, JUST A TEMPLATE!
        }\n";
	close(FILE);
	}
}

sub find_match
{
	open FILE,">>/$PREFIX/etc/objects/hosts/new_hosts.cfg" || die $!;
	my $host_cfg = `grep $ind_record[0] $path_to_hosts`;
		if($host_cfg !~ m/$ind_record[0]/g){
		mailer($ind_record[0],$ind_record[4]);
		print FILE "\#define host{
        \#use                     new_found_hosts; Name of host template to use
        \#host_name               $ind_record[0]
        \#address                 $ind_record[4]
        \#}\n";
		}
	return 0;
}	

sub mailer
{
	open FILE2,">>/$PREFIX/libexec/new_hosts"|| die $!;
	print FILE2 "$ind_record[0]\t$ind_record[4]\n";
	close(FILE2);
}

sub send_mail
{
	`/usr/bin/mailx -s "Nagios Found New Host"  $email < "/$PREFIX/libexec/new_hosts" `;
}
exit 0;

new_hosts.cfg

# Generic host definition template - This is NOT a real host, just a template!
define host{
        name                            new_found_hosts    ; The name of this host template
        notifications_enabled           1       ; Host notifications are enabled
        event_handler_enabled           1       ; Host event handler is enabled
        flap_detection_enabled          1       ; Flap detection is enabled
        failure_prediction_enabled      1       ; Failure prediction is enabled
        process_perf_data               1       ; Process performance data
        retain_status_information       1       ; Retain status information across program restarts
        retain_nonstatus_information    1       ; Retain non-status information across program restarts
	check_period			workhours
	check_interval			3
	retry_interval                  1
	max_check_attempts              2
        active_checks_enabled           1
        check_command                   check-host-alive
	notification_period             workhours
	notification_interval           15
	notification_options            d,r
	contact_groups                  admins
        register                        0       ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL HOST, JUST A TEMPLATE!
        }

Enjoy and if you have any questions let me know. The code is here and not in zipped format due to ignorant individuals who find it funny to hack into sites and destroy valuable information designed to help the community.

Mike Kniaziewicz, MIS