Syslog and SNMP

h6. Contents

This section covers Message Processing for GroundWork Monitor.
{toc:minLevel=4|maxLevel=4|printable=false}

h4. Syslog Processing

!mp_syslogprocessing.gif|align=right! The Syslog processor is a module of open source packages that will enable remote syslog messages to be forwarded to the GroundWork server and post error messages on the operator console when predefined errors occur. The syslog process includes the following components and is displayed in the diagram.

The syslog-ng package is installed on the GroundWork server. It allows remote systems to use the syslog facility to direct log messages to a directory on the GroundWork server. Configuration files will define which log messages are directed to GroundWork and which programs to call when those messages are found.

A plugin will send event messages to the Foundation LogMessage table. An operator can view a history of log messages by using the Console application. An application type of SYSLOG is predefined in Foundation so the operator may view all SYSLOG messages in the Console. GroundWork's SYSLOG Service Profile which includes Nagios SYSLOG trap plugins executes the following process:
* Reads the designated syslog file on a scheduled basis. Each time the plugin is called, it will read from the last position to the end of the file.
* Compares each line with a match file containing regular expressions. If the regular expression matches, an exception condition is flagged.
* Sends the exception event to be logged in the Foundation database.
* Queries Foundation for number of SYSLOG exception events associated with each Host. It reports the number of exception conditions in the Service output and will set the Service status to the highest severity of the matched message.
* Sends the last matched message to a passive Service named {{SYSLOG_last}}. This Service can be used for Notification or event handling.
* Configuration definitions for the summary Service and the passive Service check.

h4. SNMP Trap Messages

h5. Support for SNMP Polling

GroundWork Monitor offers direct support for SNMP polling and SNMP traps.

In simple terms, SNMP polling allows GroundWork Monitor to query network devices for common operational statistics such as memory usage and network traffic levels, while SNMP traps allow the network devices to unilaterally notify GroundWork Monitor of critical events whenever they occur.

For example, SNMP polling can be used to read the current disk utilization levels on a device every few hours, with this data being used to monitor or chart long-term consumption trends, and possibly being used to generate an alarm if the usage levels are determined to be abnormal. However, if a critical disk shortage suddenly appears in between these polling intervals, this usage scenario could allow the disk shortage to go undetected for several hours. In that kind of situation, the struggling device would best be served if it could unilaterally transmit an SNMP trap message that indicated the error condition immediately.

h5. How this Works

All SNMP messages are transported by using the User Datagram Protocol (UDP), and this includes SNMP queries and unsolicited trap messages alike. The use of UDP as a transport protocol has several advantages, but it also imposes fairly strict restrictions on the size of the message that can be sent. In order to ensure that an SNMP message will fit within the size constraints, the SNMP protocol uses a variety of encoding and compression techniques that keep the messages relatively small. However, this also means that the messages themselves are binary, and are not readily decipherable with the naked eye.

In particular, SNMP messages do not contain the full text message associated with an event, but instead uses a message ID that is uniquely associated with the underlying message. Meanwhile, information about the event (such as the severity of the event, and the full-text string message) is stored in a dictionary file called a Management Information Base (MIB), which is independently maintained by the message receiver. This model allows the sending station to simply provide the ID value in the original message, while leaving it to the receiver to map the ID value to an actual event message that can be displayed to a human operator.

For example, an SNMP management station might record an event that says "Link up on interface 2" after a network interface has come online. However, the underlying SNMP message will only contain the fixed identifier value for that event - along with the variable value for the interface that just became active - but does not contain the full "Link up" text message. Instead, the trap sender expects that the recipient system will be able to associate the code value with a local MIB and display the correct message, with the variable data being substituted into the proper location.

Along the same lines, SNMP messages are also able to refer to object IDs outside their own scope, as long as the other IDs are correctly referenced in the dictionary. For example, the numeric value for the "Link up on interface 2" message discussed above uses a numerical syntax that is described in the generic standard messages. As such, the trap dictionary for this event does not need to define the syntax rules for that variable value, but instead can simply reference the parent definition.

Taken as a whole, this approach allows the messages and dictionary files to be very small and efficient, but it also results in making the message data opaque to human users. Therefore, in order for the data to be usable by a human, this model requires that the SNMP management station maintain a dictionary of trap messages and their ID assignments, with the incoming messages being mapped to their full-text event message, with the message variable data being decoded and applied where needed.

h5. The SNMP Trap Software Architecture

By default, GroundWork Monitor provide a basic SNMP trap handler that captures a handful of common messages, which are then stored in the GroundWork Foundation database for viewing through various applications, and which are also fed to Nagios for the purpose of triggering notification messages.

More specifically, GroundWork Monitor uses the following components to receive, process, and store SNMP trap messages as they arrive from the network:

* Incoming SNMP traps are received by the {{snmptrapd}} daemon that is provided with the Net-SNMP software package. {{snmptrapd}} provides the network listener that is needed to receive incoming SNMP trap messages from the network.
* Once a trap message has been received, {{snmptrapd}} passes the incoming message to the snmptthandler program that is part of the SNMP Trap Translator software package, which in turn dumps the trap message into a spool file. This allows {{snmptrapd}} to dispose of the messages quickly.
* Periodically, the snmptt daemon included with the SNMP Trap Translator package examines the spool file for new trap messages, converts them into structured text, and then simultaneously passes the output to the GroundWork Foundation database and the Nagios NSCA passive interface. This conversion process allows the SNMP messages to be processed the same as other kinds of textual event messages.
* Trap messages that are received by the Foundation database listener are correlated with the original host device and then recorded as SNMPTRAP event messages. Once the database has been updated, the trap data can be displayed by the operator using the global Event Viewer or the host-specific Status applets.
* Meanwhile, trap messages that are received by Nagios through the NSCA interface are stored in the Nagios command file until the next check interval. When these passive checks are subsequently processed by Nagios, the incoming messages will be correlated with the original host device and processed as events for the {{snmptraps_last}} service associated with that host. From there, the messages can then be used to perform whatever actions are available in Nagios, such as generating notification messages.

Taken as a whole, this design allows GroundWork Monitor to perform as a comprehensive SNMP monitoring and management station, once the individual software components have been configured.

{Note}Prior versions of GroundWork Monitor Professional used a somewhat different architecture with different software components. If the SNMP trap support from a previous version of GroundWork Monitor is already in use, the older configuration files will need to be converted before those customizations will perform correctly with version 5.2 and beyond. GroundWork Monitor provides a conversion script in {{/usr/local/groundwork/tools/snmp_mibs/upgrade_snmptt_conf.sh}} that will perform this process somewhat automatically.{Note}

h5. The SNMP Trap Listener

The configuration settings for the {{snmptrapd}} daemon bundled with GroundWork Monitor are stored in {{/usr/local/groundwork/common/etc/snmp/snmptrapd.conf}}. The default configuration file contains the minimum directives that are needed to capture SNMP traps and pass them to {{snmptthandler}} for processing, but additional configuration directives may also be added if they are needed.

{Note}The default configuration file does not define any filters, so all incoming trap messages will be intercepted in the default setup. If you need to limit SNMP trap handling to specific hosts or a particular SNMP community, you will need to edit this configuration file accordingly. Otherwise, you can simply instruct your network devices to begin sending SNMP traps to the GroundWork Monitor server, and the messages will immediately be received and processed without any local configuration required.{Note}

h5. The SNMP Trap Translator

The raw form of an SNMP trap message uses an opaque binary syntax that is not directly usable by GroundWork Monitor or Nagios. For that reason, GroundWork Monitor relies on the open source SNMP Trap Translator (SNMPTT) package to convert the raw SNMP traps into structured textual messages that can be manipulated more easily.

Specifically, {{snmptrapd}} passes incoming SNMP traps to a program called {{snmptthandler}}, which in turn stores the trap messages into an incoming spool file. Every few seconds, an independent {{snmptt}} daemon reads the spool file for new trap messages, converts the messages into structured text, and then passes the output to the GroundWork Foundation database and the Nagios NSCA agent through the appropriate local interfaces.

SNMPTT uses multiple configuration files to determine how trap messages should be converted and relayed. First and foremost, {{/usr/local/groundwork/common/etc/snmp/snmptt.ini}} contains the global options that are needed for the {{snmptt}} daemon to operate as expected (such as database connection and logging options), and also contains pointers to subordinate trap-specific configuration files that tell the {{snmptt}} daemon how to dispose of specific trap messages.

The actual translation and disposal process is controlled by trap-specific configuration files. For example, {{/usr/local/groundwork/common/etc/snmp/snmptt.conf}} contains the processing directives for a handful of universal SNMP trap messages, and GroundWork Monitor also includes predefined translation files for a few specific devices which are typically stored in their own configuration files (the {{CISCO-GENERAL-TRAPS.conf}} configuration file contains mapping information for SNMP traps that are commonly found with Cisco network equipment, and so forth). Each of these translation-specific configuration files are already listed in the {{snmptt.ini}} control file, and will be used to process the corresponding SNMP trap messages.

If you need to capture the full trap message and severity details for SNMP traps that are not already defined, you must extract this information from the appropriate SNMP MIB files and add the trap messages to one of the control files (along with information that tells the {{snmptt}} daemon where to send the messages). GroundWork Monitor provides a variety of tools to simplify and automate portions of this process, although some manual intervention is also usually required. Refer to the Adding New Trap Definitions section below for information on how to add new trap translation definitions.

h5. GroundWork Foundation SNMP Trap Adapter

When the {{snmptt}} daemon needs to pass a converted trap message to the GroundWork Foundation event database, it passes the output to the Foundation network listener on TCP port 4913 on the loopback interface (127.0.0.1), and flags the data as an SNMPTRAP event with an appropriate severity level. In turn, the SNMPTRAP Foundation adapter examines the data and adds the event to the GroundWork Foundation LogMessage database table.

{Note}SNMPTT is configured by default to send all trap messages to the GroundWork Foundation listener, including unknown SNMP trap messages that have not been listed in {{snmptt.ini}} or a subordinate configuration file. These messages will be shown with an "unknown" severity level, and will show "unknown trap" in the message text. Refer to the Adding New Traps section below for information on adding new trap definitions to the {{snmptt}} configuration.{Note}

By default, the SNMPTRAP adapter will consolidate duplicate event messages that are received in sequence. If a trap event arrives that matches the consolidation criteria, the duplicate event message is discarded, the "count" field of the existing record is incremented, and the "last insert date" field is updated. Specifically, the consolidation feature looks at the following fields to determine if a record is a duplicate, and if all fields match then the duplicate is discarded:

* OperationStatus
* Host
* Severity
* IP Address
* MonitorStatus
* Event_OID_numeric
* Event_Name
* Category
* Variable_Bindings

An exception to the above is if the Monitoring Status of the last event is different than the Monitoring Status of the incoming event. In that scenario, a new event message will be created. This rule guarantees that console messages sorted by chronological order will always show the current status above previous status.

You can disable trap consolidation if you wish (thereby having a new event record be created for every SNMP trap). You can also change the fields that are used for the consolidation service if needed. The fields that are used for the consolidation function are stored in the SNMPTRAP entry of the ConsolidationCriteria table of the GWCollageDB database.

Once the event records have been inserted into the event database, the trap messages can be viewed from the Event Console application (selecting the SNMPTRAP option in the left navigation tree will display only SNMP trap events, and filter out other messages). Recent SNMP trap messages that have been correlated with a host can also be viewed in the Status application by selecting that host and looking under the "Console" portion of the host status screen.

h5. The Nagios SNMP Trap Handler

The {{snmptt}} daemon also passes incoming trap data to Nagios via the NSCA passive interface on TCP port 5667 on the loopback interface (127.0.0.1), and flags the data as a passive check result for the {{snmptraps_last}} service on the originating host. From there, Nagios will periodically process the incoming passive check queue and attempt to correlate the incoming events with the {{snmptraps_last}} service definition associated with the originating host.

More specifically, the {{snmptt}} daemon provides the translated trap message, and also provides the Nagios-specific result codes that indicate whether the trap is benign, a warning, a critical event, or an unknown message. In turn, Nagios is able to use the trap severity to determine when to initiate notifications and perform other actions, and can also display the trap text in the service status field.

{Note}SNMPTT is configured by default to send all trap messages to the NSCA listener, including unknown SNMP trap messages that have not been listed in {{snmptt.ini}} or a subordinate configuration file. These messages will be shown with an "unknown" severity level, and will show "unknown trap" in the message text. Refer to the Adding New Traps section below for information on adding new trap definitions to the {{snmptt}} configuration.{Note}

In order for this process to work, the Nagios NSCA passive interface must be active, the originating host must be defined in Nagios, and the originating host entry must have the {{snmptraps_last}} service definition associated with it.

{Note}The Nagios NSCA passive check interface is not active by default, and must be started before the SNMP trap messages can be received. The {{/etc/init.d/nsca}} {{init}} script is provided for this purpose. This init script should be included with the system init scripts if the NSCA service will always be needed.{Note}

{Note}The {{snmptraps_last}} service definition is not included in the default set of Nagios service definitions, although the associated service profile is provided with GroundWork Monitor.{Note}

To use this service definition, perform the following steps:

# Select the Configuration application from the main GroundWork Monitor menu.
# Select Profiles.
# Select Profile Importer. This will result in a list of all of the available host and service profiles being displayed in the main window.
# Locate the entry for {{service-profile-snmp-traps.xml}} in the list of available profiles, and activate the checkbox next to it.
# Scroll to the bottom of the main window, and click the Import button. This will result in the {{snmptraps_last}} service definition being installed into the current configuration.

Once the above steps have been completed, the {{snmptraps_last}} service definition will be available for use, and can be associated with any defined host.

h5. Adding New Trap Definitions

As was discussed earlier, the SNMP Trap Translator toolkit is responsible for converting the incoming binary trap messages into structured text, and for passing the decoded messages to the Foundation database and the Nagios NSCA passive interface. When this has been properly configured, the {{snmptt}} daemon is able to generate the correct fully-formed textual message, and can also associate an appropriate severity level for a trap message. However, the SNMP Trap Translator is only preconfigured for a few common trap messages, and in all other cases the {{snmptt}} daemon will not be able to provide an appropriate severity level, nor will it be able to provide a meaningful status text message (it will use a generic "unknown" trap message instead).

If you need to capture the full trap message and severity details for SNMP traps that are not already defined, you must extract this information from the appropriate SNMP MIB files and add the trap messages to one of the control files (along with information that tells the {{snmptt}} daemon where to send the messages). GroundWork Monitor provides a variety of tools to simplify and automate portions of this process, although some manual intervention is also usually required.

In particular, the {{/usr/local/groundwork/tools/snmp_mibs/convert_mib.sh}} script will read through a specified MIB file and locate any trap definitions contained therein. If there are no problems with the MIB file (such as unsatisfied dependencies or syntax errors), the convert_mib.sh script will create a local .conf file for use with {{snmptt}} that contains all of the trap definitions and the options that are needed for {{snmptt}} to correctly process and relay those trap messages. From there, the output .conf file can be added to the {{snmptt.ini}} control file, and the new trap messages will be processed after the {{snmptt}} daemon has been reinitialized. However, if the original MIB file cannot be converted (possibly due to unsatisfied dependencies or syntax errors), additional steps will need to be taken before the SNMP traps can be captured.

# The first step in this process is to locate the SNMP MIB file that contains the trap messages you want to capture and convert. Most equipment vendors provide the necessary MIB files with their installation software. There are also a variety of third-party MIB repositories across the Internet that can be helpful for locating MIB files.
# Once the MIB file has been obtained, use the {{/usr/local/groundwork/tools/snmp_mibs/convert_mib.sh}} script to automatically convert it into a {{snmptt}} configuration file by specifying the MIB file name as the only parameter. If the conversion operation is successful, a new file will be created in the local directory with same name as the input file, with an extra .conf suffix at the end.
\\
\\
For example, the figure below shows the command {{convert_mib.sh /tmp/UCD-SNMP-MIB.MIB}}. This command causes the convert_mib.sh script to read in the {{/tmp/UCD-SNMP-MIB.MIB}} file, locate the SNMP trap definitions in the MIB file, and then generate a SNMP Trap Translator configuration file called {{UCD-SNMP-MIB.MIB.conf}} in the current directory. In that example, the output file contains definitions for two SNMP traps, which are {{ucdStart}} and {{ucdShutdown}}, respectively.
!mp_newtrapdef.gif|hspace=5px|align=left!
{Note}The conversion script will append data to the output file instead of overwriting it. If you want to completely regenerate an existing configuration file, you must delete the existing configuration file before running the script again.{Note}

h5. Resolving MIB Dependencies

Almost all MIB files are somewhat dependent on other MIB files for certain types of data. For example, an SNMP trap may reuse a universal data-type that is defined in another MIB file, and in that scenario the trap definition will not be usable until the other MIBs have also been made available to the MIB conversion utility.

GroundWork Monitor includes several common dependency MIBs already, and in many cases the dependency issue will not be noticed. For example, the screen capture in the preceding section shows the {{convert_mib.sh}} script successfully processing the UCD-SNMP-MIB.MIB file, even though the MIB file has several dependencies. However, this is only possible because GroundWork Monitor already includes the dependency MIBs and which are therefore already available to the conversion utility.

If the conversion script fails due to one or more missing dependency MIBs, you will need to determine which MIB files are missing, and then install them in the appropriate location. This can be accomplished in the following way.

You can also review the MIB file by opening it in a text editor. The dependencies will be shown as lines of text beginning with the word "IMPORTS", followed by a list of objects to be imported, then a "FROM" clause followed by the name of the source MIB. For example, the following block of text from the MIB file for a Dell PowerConnect 3248 switch, and refers to multiple subordinate MIBs:

{code}IMPORTS
MODULE-IDENTITY, OBJECT-TYPE, Unsigned32, Integer32, IpAddress,
NOTIFICATION-TYPE, internet, Counter32
FROM SNMPv2-SMI
DisplayString, RowStatus, TruthValue
FROM SNMPv2-TC
EnabledStatus
FROM P-BRIDGE-MIB
BridgeId, Timeout, dot1dStpPortEntry, dot1dStpPort
FROM BRIDGE-MIB
PortList
FROM Q-BRIDGE-MIB
ifIndex
FROM IF-MIB;{code}

In the example above, the MIB file has dependencies on SNMPv2-SMI, SNMPv2-TC, P-BRIDGE-MIB, BRIDGE-MIB, Q-BRIDGE-MIB, and IF-MIB. Each of these MIB files will be needed before the SNMP trap messages in the original MIB can be converted for use by {{snmptt}}.

Once you have determined the dependency MIBs, you must locate the appropriate MIB files and add them to the MIB repository in GroundWork Monitor.

For clarification purposes, it is important to understand that the {{convert_mib.sh}} script is a front-end to the {{/usr/local/groundwork/common/bin/snmpttconvertmib}} Perl script that is provided with the SNMP Trap Translator package, which ensures that the appropriate options and command-line parameters are used when the configuration file is generated. However, the {{snmpttconvertmib}} script uses the {{/usr/local/groundwork/common/bin/snmptranslate}} utility that is provided with the Net-SNMP software package for the task of decoding the canonical MIB data. As such, the Net-SNMP software package is ultimately responsible for locating the dependency MIBs when the traps are first converted to text.

All of the Net-SNMP utilities use a common cache directory for common MIB files, which is the {{/usr/local/groundwork/common/share/snmp/mibs}} directory. Any MIB file that is stored in this directory is available to all of the Net-SNMP utilities for the purpose of converting and displaying MIB data in text form, and this includes the one-time conversion that is utilized whenever a {{snmptt}} configuration file is generated.

Therefore, in order for a dependency MIB to be recognized by the conversion script, the dependency MIB file must be added to the {{/usr/local/groundwork/common/share/snmp/mibs}} directory. Once the file has been copied to that directory (and the appropriate file permissions have been defined), the conversion script can be called again, and the dependent MIB files will be located and automatically used when the {{snmptranslate}} utility is ultimately called upon to generate the trap text.

By default, the {{/usr/local/groundwork/common/share/snmp/mibs}} directory contains several dozen common dependency MIB files. In addition, GroundWork Monitor also includes several other common MIB files under the {{/usr/local/groundwork/common/share/mibs}} directory tree (note that this is a different directory than the Net-SNMP cache directory), and those MIB files can be copied into the Net-SNMP cache directory as they are needed. If a MIB file has additional dependencies that are not provided by GroundWork Monitor, you may need to locate the dependency MIB files on the equipment vendor's support site or on an Internet MIB repository site.

Using the Dell PowerConnect 3248 MIB example described above, we can tell that the MIB file has dependencies on SNMPv2-SMI, SNMPv2-TC, BRIDGE-MIB, P-BRIDGE-MIB, Q-BRIDGE-MIB, and IF-MIB. Furthermore, the Net-SNMP cache directory already contains the MIB files for SNMPv2-SMI, SNMPv2-TC, and IF-MIB, but does not contain the BRIDGE-MIB, P-BRIDGE-MIB or Q-BRIDGE-MIB MIB files. However, those files do exist in the {{/usr/local/groundwork/common/share/mibs/ietf}} repository. As such, they only need to be copied into the {{/usr/local/groundwork/common/share/snmp/mibs}} directory, at which point they will be available to the {{snmptranslate}} utility whenever the {{convert_mib.sh}} script is next executed.

Note that there can sometimes be multiple nested levels of dependencies, and it may be necessary to copy more MIB files into the Net-SNMP cache directory than originally expected. For example, Q-BRIDGE-MIB has a dependency on RMON2-MIB, which in turn has a dependency on TOKEN-RING-RMON-MIB. In order for the PowerConnect 3248 MIB file described above to be processed without errors, all of these MIB files would need to be added to the Net-SNMP cache directory.

h6. Verifying MIB Syntax

Many SNMP MIB files contain a variety of errors, some of which are relatively benign, while others are sufficiently problematic that {{snmptt}} will be unable to determine how to translate the MIB. In the latter case, this may require you to locate a different version of the MIB file, or to perform some editing of the file itself (this should not be done except in extreme cases - SNMP MIB definitions use a very complicated notation format, and it is very easy to make a bad situation much worse).