About the Notification and Escalation Subsystem

1.0 Subsystem Overview and Roadmap

1.1 Notification Subsystem Overview

Since inception GroundWork has relied upon the notification and escalation functionality of Nagios to notify and escalate alerts to contact groups and contacts. With the introduction of Cloud Hub and the associated RESTful API GroundWork has introduced a free standing Notification and Escalation Subsystem which no longer requires the use of Nagios to alert contacts and contact groups. Cloud Hub bypasses Nagios for several reasons one of which is to permit changes to the server virtual infrastructure to be made fluidly and automatically with alerts no longer needling to be processed by a batch configuration commit process. Secondly, bypassing Nagios for alerts, avoids processing overhead associated with Nagios that in some scenarios avoids a capacity limitation.

A free standing Notification and Escalation Subsystem also permits changes to notification and escalation schedules to be made in run time by roles having lower privileges than the system administration role which makes the system easier and more flexible to operate. NoMa also incorporates typical business rules and conditions into its user interface that are easier to understand and configure. Finally, by removing notifications and escalations from the Nagios Configuration System as part of the multi-release re-engineering of that subsystem, we simplify the task of making that subsystem fully multi-tenant.

For the present release, notifications and escalations using Nagios remain available for configuration and maintenance as shown by the dotted line, as was the case before the introduction of NoMa at Release 6.7. This permits customers that have made investments in time or scripting for Nagios alerts to continue to use these methods. Documentation about the use of Escalation Trees to standardize this method is described in the Monarch documentation. Alternatively all versions of GroundWork since Release 6.7 can be reconfigured so that Nagios alerts are sent directly to NoMa which is configured to handle them, in order to gain the benefits described above.

As shown, alert flow coming from Cloud Hub, Cacti since Release 7.0.2, and from the Log Hub that is currently in beta testing is processed by unique feeders that send alerts to the REST API for the foundation database. In turn, NoMa subscribes to alert messages via the REST API for NoMa to perform notifications and if needed escalations. In accordance with the configuration input to NoMa.

The data flow diagram showing the relationship between Nagios, NoMa, and the GroundWork data management subsystem is shown in the following figure.

Figure: Notification and Escalation Subsystem Simplified Data Flow

1.2 Subsystem Roadmap

There are additional integration steps planned for this subsystem. In particular:

Moving the NoMa database content and structure into the Foundation Database such that data related to Notifications and Escalations can be made available for reporting, dashboards, and views.
Using the existing Nagios feeders to communicate with NoMa instead of the direct feed shown on the solid line above.
Updating the U/I look and feel for NoMa so that it is consistent with that of the remainder of the product.
Accessing the contact information from the Portal user data store in order to reduce duplicate data entry for contacts.
Adding business rules and conditions as requested by customers.

2.0 About NoMa

In this section we take a look at the NoMa application, its functions, and how the subsystem operates. Notification Manager or NoMa makes managing notifications easier. It lets you create and modify contacts, standard notifications, escalations, schedules and vacations for your monitoring system through a web interface available in run time and also easily forward them onward as required. A more detailed look at configuration is offered in Notification Configuration which is part of the NoMa document set in Bookshelf.

2.1 NoMa schema

By using the NoMa front-end, hosts and services can be assigned to notifications. When NoMa receives notifications either directly from Nagios or indirectly through the GroundWork RESTful API, it searches for matching host and service definitions in its database. If a matching setup (configured notification) has been found, escalation levels, receivers and methods are determined and notifications will be sent. The following diagram from the NoMa documentation has been modified to reflect the way the product is employed within GroundWork Monitor.

Figure: NoMa Simplified Data Flow

2.2 About NoMa and GroundWork Monitor

NoMa uses a notifier script which takes notifications from Nagios to process them as defined via the NoMa front-end. The addition of NoMa to the GroundWork platform provides an alternative way to configure and use alerting on state changes that is more flexible than the traditional Nagios Notification and Escalation method. NoMa requires certain libraries and perl modules to be present, all of which have been included in the build.

Standard NoMa
The standard NoMa daemon is started from init and displayed to the user as a web app from Apache, thus there is an init script and an apache configuration file. Users are authenticated in the browser via local Apache, ldap, or other method. The NoMa daemon keeps the record of how and when it should produce alerts in a database which can be mysql or sqlite3. NoMa presents to the user a way to select the conditions for making a notification by choosing from a list of variables that exist in Nagios/Icinga including Recipient (contact), Host, Service, Hostgroup, and Servicegroup.

The selections are made according to a lookup in the NDOutils database which must be present as the event broker for Nagios. The combination is stored in the NoMa database as one of many filters, also called Notifications. A script is provided for inclusion with Nagios as the notification command that will be triggered for every host or service you desire to be controlled via NoMa. You have to associate that command with a contact, that contact with a contact group, and the contact group with every object that will be using NoMa to alert, along with the appropriate notify conditions and time periods. On the occasion of some state change for any such, this notification transmits the specifics to NoMa daemon by calling the script; the daemon uses the specifics passed in the script compared to the stored filters in the NoMa database and on a match sends out a NoMa message to the contacts named in that matching filter. NoMa sends the message using a different script, one of several available for choice in the filter setup.

NoMa in GroundWork Monitor
GroundWork has made these changes to the way NoMa is integrated into GroundWork Monitor:

Administration of notifications, contacts, contact groups, holidays, timeframes, and methods via secure web front-end user interface.
Easily set up host and service definitions using wildcards.
Authentication via the GroundWork single sign on and role based access controls (JOSSO and JBoss Portal).
Viewer for the notification log.
Filter for non-privileged users which can only view and edit their "own" notifications.
Notification plugins are easily created to extend the functionality.
Apache configuration also adds a pointer to a distinct php instance, which has a php.ini stored in the same directory (for ease of locating it). This php.ini and the php that is called differs from the main one in GroundWork Monitor by including the yaml.so library. That lib is not compatible with our Java aware apps so we have a separate copy for NoMa.
The NoMa daemon startup is included with the GroundWork control script rather than being a stand alone startup. We pay attention to the NoMa daemon generated Process ID and use that in the normal way to status and stop the process.
The sqlite3 database choice is enforced. This is a lightweight file based engine suitable for the low volume use case. This is where NoMa keeps the Notification settings and other configuration choices.
The NoMa code is adjusted to use not the NDOUtils database for looking up filter conditions, but instead a View of the Foundation database. This means that hosts, hostgroups, services and service groups generated anywhere in the system can feed notifications, and that there is no sole dependence on Nagios for notifications. The Recipient choice is allowed to be null, or to be a choice from the NoMa Contact list, and we expect you will use the same user name (example noma) in the NoMa contact, as the contact you create in Nagios for the forwarding. It is OK to be null.
The ability to select a Nagios contact as a condition of the Notification filter from NoMa feature is lost. This element is not present anywhere but in Nagios and Monarch. It is only significant for those who must rely on Nagios settings. If this is necessary to your use case then you must import the contacts that you use in Nagios, into NoMa so they will be available for your choice.
Preview of host and services has been deactivated for now.

2.3 NoMa/Nagios configuration process

To use NoMa as an alerting agent for Nagios then you must perform the following.

In the NoMa application

Add the contacts that you wish to use as Recipient names in the notification setup, from the Nagios configuration.
Add the contacts that you wish to be alerted when NoMa sends out an alert. These may be different than the Nagios set.
Create the notifications in NoMa, associating objects (chosen from Foundation) with contacts, time periods, and conditions for alerting.

In Nagios

Add the contact to Nagios that you will associate with the forwarding of the alerting.
Associate the alert via NoMa script as a notification command for hosts and for services with that contact.
Add that contact to the contact groups according to your needs.

3.0 Operating NoMa

The NoMa user interface includes screens for Configuration Mode and for Operation Mode. Notifications, Contacts, Holidays, Contact Groups, Timeframes, and Methods are used to maintain the configurations determining who gets notified and when. The Logs display is used in Run Time to monitor the notification and escalation process.

3.1 NoMa configuration User Interface

Figure: NoMa U/I

3.2 Notifications configuration

In this display, the operator creates a notification by associating contacts, contract group, time periods with the host groups, hosts, and services generating the alerts. Its use requires the prior creation of contacts, contact groups, and time frames if these elements are to be used.

Figure: Notifications U/I

3.3 Contacts configuration

Contacts are definitions which contain individual settings defining who should get notified in the event of a problem on your network. Contact definitions also indicate which notification options will be used for the contact based on the selected notification definition. The Contacts user interface is where you can add, modify, and delete notification contacts. The directives consist of contact properties: Fulll Name, Username, Timezone, Timframe, Suppress multiple alerts, and Contact addresses information, and Holiday.

Figure: Contacts U/I

3.4 Holidays configuration

Holidays can be created from the Contacts or Holidays tabs and are applied within the Contacts configuration. A holiday is named and given a start and end date and a timeframe and assigned a previously defined contact.

Figure: Holidays U/I

3.5 Contact Groups configuration

Contact groups are definitions of one or more contacts for the purpose of sending out alert/recovery notifications to one or more contacts. Contact definitions can be grouped into contact groups typically by area of expertise or geographic location. For example you might have one contact group called network-administrators and perhaps another contact group called baltimore-support. Then, when a host or service has a problem or recovers, NoMa will find the appropriate contact groups to send notifications to and notify all contacts in those contact groups. This display is used to create Contact Groups.

Figure: Contact Groups U/I

3.6 Timeframes configuration

Timeframes are lists of times during various days that are considered to be valid times for notifications and service checks. They indicate when host and service notifications can be sent out to contacts and when notification escalations can be used. Timeframe information is defined in the Timeframes tab and applied within the Notifications tab.

Figure: Timeframes U/I

3.7 Methods configuration

Notification methods are configured from the Methods page. These define the information that is used in each method of notification (e.g. Voice uses the command voicecall and the contact field phone).

voicecall - Voice alerts are generated with an Asterisk (or Starface) soft PBX.
sendsms - SMS alerts are sent via an attached SMS-capable modem using smstool3 or an iSMS/SMSFinder hardware device.
sendemail - Emails are sent using the standard system mailer - you may need to configure your mail relay to accept mails from the system.
growl - Growl notifications are sent using an included script. It requires the Perl module Net::Growl and by standard it uses UDP 9887 by default. After install, allow notifications from network, optional require password from LAN.
Server only requires Perl module Net::Growl as stated earlier.
Client, depending on OS, requires an extra client:
- Mac: http://growl.info
- Linux: http://mattn.github.com/growl-for-linux/
- Windows: http://www.growlforwindows.com/gfw

Figure: Methods U/I

3.8 Logs display

The log viewer, accessed via the Logs tab, shows the NoMa notification log which is accessible during runtime. It displays all of the alerts in the order they were generated with relevant status information.

Figure: Logs view