GWME-7.1.1-10 - Updated NoMa fixes

This patch supersedes the "GWME-7.1.1-9 - NoMa fixes" patch, and includes all the fixes that were included in that patch.

Problem

NoMa has been found to have a few bugs affecting the reliable delivery of notifications.

Solution

This patch rolls up all the available NoMa-related fixes into one patch for the GWME 7.1.1 release. Some NoMa files are replaced, the config/foundation.properties file is augmented with a new configuration option, and unique-ID values are adjusted in the noma database.

The new option (fas.executor.interrupt) controls how long the Java thread that runs the alert_via_noma.pl script for CloudHub-related notifications can run. Field experience shows that the historical hardcoded timeout has been too small for reliable operation in the context of NoMa. Exposing this parameter in the config file allows it to be adjusted if necessary. The default in the config file is now set an order of magnitude larger, which should be sufficient to prevent problems even on large, heavily-loaded systems.

Installing

You may install this patch directly over the GWME-7.1.1-9 patch
If you have previously installed the "GWME-7.1.1-9 - NoMa fixes" patch you do not need to uninstall it before installing this patch. You can simply install this newer, more comprehensive patch on top of the older patch.
  1. Download the patch file tar archive to, for example, the /tmp directory.
    Name Size Creator Creation Date Comment  
    ZIP Archive TB7.1.1-10.noma_fixes.tar.gz 94 kB Glenn Herteg Sep 07, 2017 14:27 MD5: dcd7859aa1c907c86fd5b7c8bb5cc178  
  2. Unroll the downloaded tar archive. The patch files will appear in the TB7.1.1-10.noma_fixes/ subdirectory. Go there and run the install script.
    cd /tmp    # or wherever you downloaded the TB7.1.1-10.noma_fixes.tar.gz tarball to
    service groundwork stop noma
    tar xvfz TB7.1.1-10.noma_fixes.tar.gz
    cd TB7.1.1-10.noma_fixes
    ./TB7.1.1-10_install
    

    The original files which are affected by this patch are first backed up, then the changes are applied, and the patch directory is adjusted to reflect the application of this patch.

  3. Bounce NoMa, to run using the replacement files. Also bounce Foundation, to pick up the non-default setting for the fas.executor.interrupt parameter.
    service groundwork restart noma
    service groundwork restart gwservices
    

Uninstalling

Uninstalling this patch must force the loss of operational data
The changes to NoMa made in this patch switch it from using externally-generated unique-alert-ID values (mostly from Nagios) to using internally-generated unique-ID values. This is necessary for NoMa to fully support CloudHub and other agents that might send in alerts. Those agents have historically not provided proper ID values, and it was deemed to be the wrong approach to have them do so. Instead, we now centralize the tracking of alerted-upon states within NoMa itself.

One consequence of this shift is that we need to alter the operational data in the noma database when this patch is installed. Of necessity, this is essentially a one-way transition. It is done to prevent collisions between historical ID values and those which will henceforth be internally generated. No record is kept of the old values that would be of any use in later trying to revert the database content. And in any case, new alerts created while the patch is in play will be following a new regime that would not have entries in such a mapping.

The upshot is that if you decide to uninstall this patch, you will probably need to clear out all the operational tables in the noma database at the same time. This will leave an open field for further operation of NoMa.

Uninstalling this patch does not revert the config/foundation.properties file
The change made to the config/foundation.properties file for this patch is quite simple, and desirable anyway for other reasons. More importantly, other patches might also alter this file, and it would be unfortunate if uninstalling this patch inadvertently rolled back some of those changes. Therefore, we leave this file alone during an uninstall of this patch. The original file is still available in the backup tarball made during patch installation, so that is one place it can be found if there is some desperate need for it. If it finds it needs to modify the file, the install process also makes a sibling config/foundation.properties.pre-TB7.1.1-10-noma file that is more convenient to reference, and can be directly compared against the config/foundation.properties file to highlight any differences in the final file contents.
  1. Go back to the patch directory, and run the uninstall script.
    service groundwork stop noma
    cd TB7.1.1-10.noma_fixes
    ./TB7.1.1-10_uninstall
    

    The backup directory will be accessed to restore the original files, and the patch directory will be processed to reflect the restoration of those files.

  2. Clear out the noma database operational tables. After this step, general setup will be left intact, but all record of previous notifications will be gone. Enter the password for the noma database when requested.
    /usr/local/groundwork/postgresql/bin/psql -U noma -d noma
    delete from escalation_stati;
    delete from notification_logs;
    delete from notification_stati;
    delete from tmp_active;
    delete from tmp_commands;
    \q
    
  3. Bounce NoMa and Foundation, to revert back to the original files and the original setting for the fas.executor.interrupt parameter.
    service groundwork restart noma
    service groundwork restart gwservices
    

Configuration

The Nagios commands to send notifications to NoMa use the alert_via_noma.pl script. The flag used to pass the incident id (-u) has historically been mis-documented, and needs to be updated to refer to the host or service PROBLEMID instead of the NOTIFICATIONID. This change is necessary for subsequent notifications on the same incident to function properly, clocking the alert-counting logic so the right contacts get sent notifications as configured. The commands below reflect not only that change, but some other refinements as well.

That said, installation of this patch switches NoMa to generating incoming-alert IDs internally. (This is controlled by the new generate_IDs option in the noma/etc/NoMa.yaml config file, which will be automatically enabled when this patch is installed.) Switching to internal generation of ID values means that the -u option value will be ignored. So we document the clean construction here for a clear understanding of what would need to be in play if Nagios were the only alerting agent and internal generation of ID values was disabled. Regardless, you should put these definitions in place to avoid any future confusion.

These are the updated commands we will be using in future GWMEE releases. You can copy/paste these definitions into your own copies of these commands in Monarch (under Configuration > Commands > Modify).