GroundWork Foundation

Contents

This section reviews GroundWork Monitor Foundation and is intended to be a developer's guide for integrating monitoring data into the Foundation Data Store. Additionally, we have included a Foundation tutorial which walks through how to develp a new Feeder and Adapter for inserting custom data streams into the Foundation framework.

1.0 Foundation Overview

Foundation is an IT management data abstraction layer and development platform. The Foundation data model allows the integration of any state, event, and performance data, independent of the Monitoring Application that produces it. This offers the possibility to store data for additional systems, including open source and commercial monitoring systems, databases, and even hardware, such as detectors or sensors. It also allows the integration of Application Monitoring data known as MBeans.

The intent of Foundation is to provide a data model that integrates the components of an IT infrastructure requiring monitoring. Flexible methods of integrating data into the data store are provided allowing different tools and applications and databases to feed data into Foundation. Foundation will normalize the data so it can be retrieved in a consistent manner. Foundation then provides various APIs to allow the normalized data to be retrieved. The Foundation package includes Nagios, as the main monitoring system, integrated with Foundation and a set of applications which use the Foundation APIs to present real-time views and reports.

The Web Service interface is a new addition to the existing Foundation Framework. The previous API components have been re-packaged to use the Web Service Interface and have not been replaced. Select the Display Diagram icon above to view the different Foundation components and their interaction.

APIs for Perl, PHP, and Web Service are part of the toolkit. The Perl API is a module called CollageQuery that allows a Perl program to retrieve data from Collage. The GroundWork PHP API provides an object library to access the Foundation database sources and the information within these sources. The library is split up into connection and query type classes. Adding a Web Service layer enables more applications to use and integrate with the existing Foundation Framework. We'll review adding a Web Service layer which enables more applications to use and integrate with the existing Foundation Framework. And, the Foundation 1.5 consolidation feature allows you to reduce the number of LogMessages by creating just one entry for messages that are alike, incrementing the Message counter, and adjusting the date fields. The actual implementation of the APIs is part of the GroundWork Monitor installation.

Programmers can use PHP, Perl, or Web Service API or third party products to:

  • Build custom displays of real-time monitoring information
  • Build custom reports from historical monitoring information
  • Monitor additional devices or systems by feeding monitoring information into the system
  • Consolidate information from disparate systems into a single view
  • Integrate heterogeneous systems by using GroundWork Foundation as an intermediary system
1.1 Deployment of Foundation

Foundation is packaged and deployed as a web application (.war) into the Jetty Servlet container. Jetty is an open-source, standards-based, full-featured lightweight servlet container implemented entirely in Java. In addition to the Foundation application the GroundWork implementation includes the following web applications:

  • GroundWork Report Server and Eclipse BIRT Viewer that allow to run and manage Eclipse BIRT Reports created with Eclipse BIRT Report Designer.
  • GroundWork JMS. A full featured persistence and topic server based on the Open Source project JORAM.
  • GroundWork Monitor includes Event Console implemented as a Web application.
1.2 Related Applications

The Monarch tool is a web application that is used to configure the Nagios system. It stores the Nagios configuration data in its own database. At this point in time, the Monarch database is separate from the Foundation database. On a monarch commit the new configuration is synchronized with Foundation.

1.2.1 Storing Nagios Data in Foundation

Since the Nagios monitoring system is integrated in this package, any information gathered by a Nagios plugin can be integrated into the system. Nagios Feeders or the Event Broker take the information from the Nagios system and inserts it into the Foundation database. The data objects contained in Foundation map closely to the Nagios objects listed below.

APIs built on top of the Foundation framework allow this information to be retrieved. The APIs allow programs to query by object and data type. Separate APIs are available for Java, PHP, and Perl programs. In addition to the provided samples, the Foundation status views (Overview and Troubleview) are built using the PHP Foundation API.

Data Objects

  • Host Groups - This includes Hosts as members.
  • Hosts - This typically represents a monitoring entity that in Nagios usually maps to physical devices. A Host entry contains one or more Service Checks.
  • Services - This typically represents a Nagios Service Check for a specific Host. A Host-Service combination is unique in the monitoring instance.

The following type of information can be retrieved:

  • Host Status - This represents the current status and attributes of Host objects.
  • Service Status - This represents the current status and attributes of Service objects.
  • Events - These are typically timestamped messages that are generated by a monitoring system or managed device. The following Nagios Events are stored in the LogMessage table.
  • Host Alerts - Events generated when a Host changes state.
  • Host Notifications - Events generated when a notification occurs based on a Host Alert event.
  • Service Alerts - Events generated when a service changes state.
  • Service Notifications - Events generated when a notification occurs based on a Service Alert event.
  • Nagios Acknowledge - Existing Host or Service Alerts are updated when a user acknowledges messages.
1.3 Foundation Architecture and Data Flow
1.3.1 Architecture

The Foundation framework consists of five main components. Select the diagram icon above to view the Foundation Components:

  • Feeders These are scripts or programs which generate a data set that is sent to the Foundation Listener. The protocol is a simple XML stream.
  • Foundation Listener This is a port or Java Message Service (JMS) Listener which receives the XML streams from various Feeders and dispatches them to data normalizers, called Adapters.
  • Foundation Adapters These Adapters are programs within the Foundation framework that apply rules and data normalization to incoming data. Each Adapter is application specific (e.g. NagiosEvent, SNMP or Syslog) and is easily added and managed with the framework.
  • Foundation Persistence Service This is a Relational Persistence layer which runs on the top of a database or a database cluster.
  • Foundation API This is documented API's for PHP and Perl, used to retrieve data from the data store.

    Figure: Foundation Architecture
1.3.2 Data Flow

The data flow of messages is unidirectional, since the Foundation Framework doesn't reply to the incoming XML streams.

Figure: Foundation Data Flow

  • Feeder
    In order to integrate data into the Foundation framework, the data generated by the source application (e.g. Nagios or Java Management Extensions (JMX) Service) needs to be read and sent as an XML stream to one of the listeners. This functionality is provided by Feeders. A Feeder can be written in any language; for example, Nagios Feeders are written in Perl. The XML output protocol is simple:
    <FeederName AttributeName='AttributeValue' AttributeName='AttributeValue' ... />

    The FeederName matches with the Adapter name, and the Attributes are just a list of name value pairs. For example, the Nagios Event Feeder XML has the following format:

    <NAGIOSLOG MonitorServer='localhost' Severity='HIGH' TextMessage='Failed to check Host' />

    The Feeder could include the logic for normalizing the data, but this is discouraged. The best approach is to have a simple and generic Feeder that reads and forwards the data to the Listener. Normalization functions are best performed by Adapters.

    The simple format of an XML element represents as well one transaction across the system. For a large load this is expensive and affects the overall message throughput since transaction carry some overhead.   The recent version of Foundation includes support for more complex messages where multiple messages can be bundled into one transaction. More details about the different adapters can be found in the "Data Integration approaches section".

  • Foundation Listener
    The Listener is a simple service, either listening on port 4913 or on a JMS topic. The incoming XML message is analyzed and forwarded to the appropriate Adapter as defined in the XML element (e.g., the Adapter that matches FeederName).
  • Adapters
    Adapters are data normalizers that apply normalization or simplification rules to the incoming XML message. For example, an Adapter could calculate the average temperature for a data feed of 10 sensors in a server room, and insert the calculated value into the data store.

    An Adapter can be used to validate incoming data for completeness. It can and should be used to reject incomplete or faulty data before it gets rejected by the persistence layer, which would affect system performance.

    Adapters are written in Java and compiled into a jar library package. The package includes a Spring assembly file which is read by the Foundation Framework at initialization time. See the tutorial later in this document for more details about the syntax of the assembly file, and how to deploy an Adapter.
1.4 Data Integration Approaches
1.4.1 Before you Start

Before you start integrating data into the Foundation data store, you need to decide the following:

  1. How to collect data from the source application and how to write the Feeder.
  2. Where the data normalization takes place (Feeder or Adapter).
  3. Whether the default fields in the data model are enough to store your data, or whether you need to add application specific properties.
  4. What ApplicationType to use for your data. The ApplicationType is a parameter that allows you to access your data using a simple filter. In GroundWork Monitor, this filter is built in to the Console application view, and will show up automatically when data is present with the application type in question.
  5. Does it make sense to bundle messages into a single transaction. Bundling would allow a higher message throughput and submit all or nothing of depended data. It adds more complexity to the feeder creating feeds. Foundation added support for this type of messages by defining an XML Schema (link to file) and a new Adapter called SystemAdapter. The recent version of Foundation uses this approach for processing Nagios Status and Event messages.
1.4.2 Supported Adapters

Foundation comes with a set of Adapters for different type of data feeds. The adapters can be classified into two different types:

  • Single Transaction messages
    The adapters of this type accept XML feed of the format:
    <ELEMENT atribute=value,.. />

    Each XML element represents a transaction. Foundation supports the following adapters.

    Adapter Type XML Comments
    NAGIOS Events <NAGIOS_LOG attribute,.. /> Events from Nagios
    Nagios Status <SERVICE_STATUS attributes,./>
    <HOST_STATUS attribute,../>
    Host and Service status updates
    SNMP Trap events <SNMPTRAP attribute,.. /> SNMP trap events coming from the SNMPTT daemon
    SNMP Trap events <SNMPTRAP attribute,.. /> SNMP trap events coming from the SNMPTT daemon
    Syslog events <SYSLOG attribute,.. /> Syslog messages from the gw_syslog plugin
    Generic Events <GENERIC_LOG attribute,../> Generic adapter that maps attributes directly to dynamic properties without checking.
    System messages <COLLAGE_LOG attribute,../> Reporting System messages to Console
  • User definable transaction messages
    The preferred way to feed data is using the new SystemAdapter which allows the sending of multiple messages of different entities (Event or Status) in a single transaction. The advantages of this approach are:
    • Higher message throughput under load.
    • The SystemAdapter uses an XML schema to validate the feed upfront. Any synatx errors are detected before being processed in the service layer.
      View XML Schema: SystemConfig.pdf
    • Wrapping dependent messages in a transaction. If any of the message fail everything will roll back and guarantee data consistency.
      For information on how to use and configure feeds that use the SystemAdmin adapter see example in the section Configuring Data Feeders in this document.
1.4.3 The Feeder and Generic Adapters

As mentioned in the earlier chapters, the Feeder has to produce an XML stream that can be sent to one of Foundation's Listener services. If the Feeder performs normalization, or if the input data is simple and matches the default data model properties, then the Feeder can send the data to the Generic Adapter. The Generic Adapter maps the sent attributes to database properties without validating the values.

A sample of how to feed data to the Generic Adapter can be found below under Creating a Feeder for LOG4J and Using the Generic Log Adapter.

1.4.4 Custom Properties

If you decide that your application needs more properties to be stored along with the default Status and Event data fields, the following steps are necessary:

  1. Add new properties and their type to the PropertyType table.
  2. Associate the properties with the EntityType such as LOG_MESSAGE, SERVICE_STATUS, or HOST_STATUS.
  3. Define an ApplicationType for your data.

In the current version of GroundWork Foundation the above database operation needs to be executed with SQL statements. These methods will be supported by the next update of the Admin Feeder, which will allow dynamic addition of properties via the input stream.

Custom properties can be inserted using the Generic Adapter, but since no consistency checking is applied, message feeds with missing properties will be rejected.

1.4.5 Recommendations
  1. For large numbers of constant data feeds, implementation of an Adapter to validate the incoming data (all required fields available, correct type) is recommended. This will ensure that any error that would cause a transaction rollback (an expensive middle layer operation) can be detected up front and rejected.
  2. Feeders should be simple and as generic as possible - collect data points and send them to the Adapter for normalization. This reduces the load of concurrently running Feeder processes, which can be inefficient, especially when written in interpreted languages such as Perl.
  3. Use the SystemAdapter whenever possible. The performance improvements and the improved message validation make the system much more robust.

2.0 Configuring Foundation

2.1 Description of the Foundation System

Foundation is a system of several loosely coupled components described below:

  • Foundation-webbapp - A core component that includes the business objects, the data persistent component (Object Relation Mapping ORM), the data normalizer components (Foundation adapters), and the Web Service API (Soap based API).
  • Foundation-JMS - The server hosting the Message Queue for incoming data feeds and a Topic server for notification.
  • Foundation-reportserver - An application to manage BIRT reports.
  • birtviewer - An Eclipse application to view reports generated with Eclipse BIRT Report Designer.

Each of the components are build as a Web Application and deployed into the servlet container (Jetty). In addition to the web applications Foundation includes the following components:

  • Nagios Feeders - The Nagios feeders read Nagios status and log files and send XML messages to Foundation. The feeder scripts are located in /usr/local/groundwork/foundation/feeder directory and are named nagios2collage_status.pl which reads the Nagios status log and updates the Status database with Host and Service status information.
  • GroundWork Web Service Plugin for Eclipse - The plugin is included in the distribution (usr/local/groundwork/foundation/eclipse) but as well bundled with the BIRT Viewer web component.
2.2 Deployment of Foundation
  • Foundation Files and Components - /usr/local/groundwork/foundation
  • Web Applications - foundation/container/webapps
  • Context Files for Web Applications - foundation/container/contexts
  • Configuration for the Jetty servlet Container - foundation/container/etc
2.3 Foundation Configuration

Foundation configuration uses properties files stored in /usr/local/groundwork/config. Changing any of these files requires a restart of gwservices as described in the table below.

Configuration File Description
db.properties Contains the database credentials for any database used by GroundWork Monitor
foundation.properties Defines runtime configuration such as the port to listen on, location and configuration of the JMS server and properties to tune the application such as size of the different thread and connection pools.
adapter.properties List of adapters (Normalizer components) used for message processing.
gwreportserver.properties Defines the location of the reports and information about the report viewer.
log4j.xml This file allows to change the level of log reporting for all Java applications. By default is set to Error only. For debugging purposes it can be set to Warning, info or debug.
2.4 Logging and Log Output

The output of the log files are defined in log4j.xml. By default all the log files for Java go to the directory:

/usr/local/groundwork/foundation/container/logs
2.5 Running Foundation

To start foundation issue the following command (needs root privileges):

/usr/local/groundwork/ctlscript.sh start gwservices

and similar to stop the service:

/usr/local/groundwork/ctlscript.sh stop gwservices
Caution needs to be applied when starting and stopping services since stopping foundation will shutdown the message listeners and the API driving the User Interface screens.

3.0 Configuring Data Feeders

The following section describes configuring the system for custom data integration and the steps necessary to setup and configure data feeds into GroundWork Monitor for status and event monitoring.

3.1 Scenario

An application, let's call it TemperatureWatcher, monitors 10 temperature sensors in different rooms of a building. At startup and shutdown the TemperatureWatcher applications sends events. In normal operations every 30 seconds the temperature of each sensor is sent to GroundWork Monitor.

Additionally, the TemperatureWatcher checks the status of the sensors. If the status is different from an OK status and the sensors do not respond an event is sent to GroundWork Monitor.

The Events generated by TemperatureWatcher should be visible in the Event Console while the temperature of each sensor should be visible in the Status application.

3.2 Setup and Configuration

Given the above specifications the following meta data needs to be generated in Foundation:

  • ApplicationType: TempWatcher
  • Custom Property for Status information: Temperature of type Double - This is the field where the temperature measurements are stored.
  • Host Group: TemperatureWatcherApps - The HostGroup needs to be created so that the TemperatureWatcher applications are visible in Status.
3.2.1 Insert Metadata

The easiest way to insert the metadata into Foundation is to stop Foundation, update SQL and restart Foundation.

  1. Stop Foundation from the command line logged in as root:
    /usr/local/groundwork/ctlscript.sh stop gwservices
  2. Update SQL data into the database:
    # psql -U postgres gwcollagedb
    gwcollagedb=# INSERT INTO ApplicationType(ApplicationTypeID, Name, Description, StateTransitionCriteria) VALUES (200,"TEMPWATCHER", "System monitored by TemperatureWatcher", "Device;Host;ServiceDescription");
    gwcollagedb=#I INSERT INTO PropertyType(Name, Description, isDouble)  VALUES ("Temperature", "", 1);
  3. Restart Foundation:
    /usr/local/groundwork/ctlscript.sh start gwservices

Configuration Data for the Application, Checks and Hostgroup are XML feeds to the TCP port 4913 using the SystemAdmin adapter for Data Normalization. The following Feed creates the necessary entries so that the system can accept monitoring data from the TemperatureWatcher application:

<Adapter Session="1" AdapterType="SystemAdmin">
     <Command Action='ADD' ApplicationType='TEMPWATCHER'>
          <Host Host='Temp-Watcher-1' Description='TemperatureWatcher'
          Device='TemperatureWatcher' DisplayName='TemperatureWatcher' />
     </Command>
     <Command Action='ADD' ApplicationType='TEMPWATCHER'>
        <Service Host='Temp-Watcher-1' ServiceDescription='Sensor_1'
           CheckType='PASSIVE' StateType='SOFT' MonitorStatus='PENDING'
           LastHardState='PENDING' />
        <Service Host='Temp-Watcher-1' ServiceDescription='Sensor_2'
           CheckType='PASSIVE' StateType='SOFT' MonitorStatus='PENDING'
           LastHardState='PENDING' />
        <Service Host='Temp-Watcher-1' ServiceDescription='Sensor_3'
           CheckType='PASSIVE' StateType='SOFT' MonitorStatus='PENDING'
           LastHardState='PENDING' />
        <Service Host='Temp-Watcher-1' ServiceDescription='Sensor_4'
           CheckType='PASSIVE' StateType='SOFT' MonitorStatus='PENDING'
           LastHardState='PENDING' />
        <Service Host='Temp-Watcher-1' ServiceDescription='Sensor_5'
           CheckType='PASSIVE' StateType='SOFT' MonitorStatus='PENDING'
           LastHardState='PENDING' />
        <Service Host='Temp-Watcher-1' ServiceDescription='Sensor_6'
           CheckType='PASSIVE' StateType='SOFT' MonitorStatus='PENDING'
           LastHardState='PENDING' />
        <Service Host='Temp-Watcher-1' ServiceDescription='Sensor_7'
           CheckType='PASSIVE' StateType='SOFT' MonitorStatus='PENDING'
           LastHardState='PENDING' />
        <Service Host='Temp-Watcher-1' ServiceDescription='Sensor_8'
           CheckType='PASSIVE' StateType='SOFT' MonitorStatus='PENDING'
           LastHardState='PENDING' />
        <Service Host='Temp-Watcher-1' ServiceDescription='Sensor_9'
           CheckType='PASSIVE' StateType='SOFT' MonitorStatus='PENDING'
           LastHardState='PENDING' />
        <Service Host='Temp-Watcher-1' ServiceDescription='Sensor_10'
           CheckType='PASSIVE' StateType='SOFT' MonitorStatus='PENDING'
           LastHardState='PENDING' />
     </Command>
     <Command Action='ADD' ApplicationType='TEMPWATCHER'>
        <HostGroup HostGroup='TemperatureWatcherApps' />
     </Command>
     <Command Action='MODIFY' ApplicationType='TEMPWATCHER'>
        <HostGroup HostGroup='TemperatureWatcherApps' >
        <Host Host='Temp-Watcher-1' />
        </HostGroup>
     </Command>
</Adapter>
Feeding Data

Any data (Status and Events) will be sent to the SystemAdmin adapter since the data normalization is done by the feeder application.

Sending an event will use the following XML feed send to the TCP port 4913. The example reports a FATAL error on the sensor_1 check:

<Adapter Session='8' AdapterType='SystemAdmin'>
     <Command Action='ADD' ApplicationType='TEMPWATCHER'>
        <LogMessage MonitorServerName='localhost'
           Device='TemperatureWatcher' TextMessage='Sensor is not responding – failed to get temperature'
           ReportDate='2008-01-23 01:45:26' Severity='FATAL' MonitorStatus='FAILED'/>
     </Command>
</Adapter>

For best performance the status of all the 10 sensors should be sent as one message to Foundation:

<Adapter Session="1" AdapterType="SystemAdmin">
     <Command Action='MODIFY' ApplicationType='TEMPWATCHER'>
        <Service Host='Temp-Watcher-1' ServiceDescription='Sensor_1'
           MonitorStatus='OK' Temperature='76.3' />
        <Service Host='Temp-Watcher-1' ServiceDescription='Sensor_2'
           MonitorStatus='WARNING' Temperature='88' />
        <Service Host='Temp-Watcher-1' ServiceDescription='Sensor_3'
           MonitorStatus='OK' Temperature='43' />
        <Service Host='Temp-Watcher-1' ServiceDescription='Sensor_4'
           MonitorStatus='OK' Temperature='67' />
        <Service Host='Temp-Watcher-1' ServiceDescription='Sensor_5'
           MonitorStatus='OK' Temperature='76.3' />
        <Service Host='Temp-Watcher-1' ServiceDescription='Sensor_6'
           MonitorStatus='OK' Temperature='66' />
        <Service Host='Temp-Watcher-1' ServiceDescription='Sensor_7'
           MonitorStatus='OK' Temperature='52.3' />
        <Service Host='Temp-Watcher-1' ServiceDescription='Sensor_8'
           MonitorStatus='OK' Temperature='50' />
        <Service Host='Temp-Watcher-1' ServiceDescription='Sensor_9'
           MonitorStatus='WARNING' Temperature='93' />
        <Service Host='Temp-Watcher-1' ServiceDescription='Sensor_10'
           MonitorStatus='OK' Temperature='66.3' />
     </Command>
</Adapter>

With this configuration Status Data is visible in Status under the HostGroup TempWatcherApp and all Events are visible in the Event Console.

4.0 Configuring Consolidation

4.1 How It Works

The Foundation consolidation feature allows you to reduce the number of LogMessages by creating just one entry for messages that are alike, incrementing the Message counter, and adjusting the date fields. Consolidation processing is applied to all incoming messages of Entity Type LogMessage where the attribute consolidation is defined (e.g. consolidation='SNMPTRAP').

Consolidation criteria is stored in the database and consists of a user definable Name and the Criteria. The criteria is a semicolon (;) separated list of LogMessage record fields or PropertyName. For an incoming message to be consolidated, all values of the fields defined in the Criteria have to match an existing record.

An incoming message must define the Name of the consolidation criteria that it will be matched against. If the consolidation tag is missing, a new log entry is created, which is the system's default behavior.

4.2 Default Consolidation

By default, the consolidation is turned on for Nagios, SNMP, and Syslog event processing. Consolidation will occur if the fields defined in the consolidation criteria match. An exception to the above is if the Monitoring Status of the last event is different than the Monitoring Status of the incoming event, a new console message will be created. This rule guarantees that console messages sorted by chronological order will always show the current status above previous status. The following consolidation criteria is defined in Foundation's ConsolidationCriteria table. An Administrator can update consolidation criteria entries to the consolidation criteria table within GroundWork Monitor Administration>Foundation>Manage Consolidation Criteria. Here you will find existing consolidation criteria and a place to enter and add new criteria.

Table: Consolidation Criteria

Name Criteria Feeder
NAGIOSEVENT Device;MonitorStatus;OperationStatus;SubComponent;ErrorType  
SYSTEM OperationStatus;Device;MonitorStatus;ApplicationType;TextMessage EventBroker
SNMPTRAP OperationStatus;Device;ipaddress;MonitorStatus;Event_OID_numeric;
Event_Name;Category;Variable_Bindings
snmptt forwarding traps to Foundation
SYSLOG OperationStatus;Device;MonitorStatus;ipaddress;ErrorType;SubComponent gw-syslog-feeder.pl


Figure: Updating Consolidation Criteria

4.3 Disable Consolidation

To disable consolidation for the Feeders shipped with GroundWork Monitor, you can modify the Nagios event log feeder, nagios2collage_eventlog.pl, to send a message to Foundation that does not specify consolidation. Change the script that contains the following line:

my $xml_message = "< NAGIOS_LOG consolidation='NAGIOSEVENT'"; # Start message tag. Consolidation is ON

To the following:

my $xml_message = "< NAGIOS_LOG "; # Start message tag. Consolidation is now turned OFF
Example

To enable consolidation, the event message needs to look like the following, followed by the other arguments:

<NAGIOS_LOG consolidation='NAGIOSEVENT'

The Consolidation Criteria in the database for NAGIOSEVENT defines the following criteria:

Device;MonitorStatus;OperationStatus;SubComponent;ErrorType

If an event is fed to the system and the consolidation criteria is defined, the system checks if any log message for the Device, MonitorStatus, OperationStatus, SubComponent and ErrorType  already exists. The system will only consolidate if:

  • just one LogMessage object matches
  • if multiple LogMessage objects match a warning is logged indicating that the criteria needs to be better defined

If a match is found an existing message the message counter LogMessage.MsgCount for an existing message will be incremented and the date fields will be updated as follows:

Database Field Change Applied
FirstInsertDate unchanged
LastInsertDate ReportDate
ReportDate System (current time)
TextMessage Updated Text message. Text Message might include the values of a check (85% disk used) that changes while the status and the type remain the same.

5.0 Data Objects and Attributes

Host Groups, Device, and Monitoring Server attributes are processed by the SystemAdmin adapter.

Foundation's data store has a flexible data format that can be expanded if needed. The Foundation data objects and attributes implemented in the Foundation package that are applicable to a Nagios-centric are listed below.

The attributes listed below are Nagios specific and apply to the application type Nagios. The Nagios Adpaters (Nagios_log (Events), Host_Status Service_Status) check for the existence of the following attributes.
Host Status Attributes
  • MonitorStatus
  • LastCheckTime
  • LastStateChange
  • isAcknowledged
  • TimeUp
  • TimeDown
  • TimeUnreachable
  • LastNotificationTime
  • CurrentNotificationNumber
  • isNotificationsEnabled
  • isEventHandlersEnabled
  • isChecksEnabled
  • isFlapDetectionEnabled
  • isHostIsFlapping
  • PercentStateChange
  • ScheduledDowntimeDepth
  • isFailurePredictionEnabled
  • isProcessPerformanceData
  • LastPluginOutput
Service Status Attributes
  • Host
  • MonitorStatus
  • RetryNumber
  • StateType
  • LastCheckTime
  • NextCheckTime
  • CheckType
  • isChecksEnabled
  • isAcceptPassiveChecks
  • isEventHandlersEnabled
  • LastStateChange
  • isProblemAcknowledged
  • LastHardState
  • TimeOK
  • TimeUnknown
  • TimeWarning
  • TimeCritical
  • LastNotificationTime
  • CurrentNotificationNumber
  • isNotificationsEnabled
  • Latency
  • ExecutionTime
  • isFlapDetectionEnabled
  • isServiceFlapping
  • PercentStateChange
  • ScheduledDowntimeDepth
  • isProcessPerformanceData
  • isObsessOverService
Event - Host or Service Alert
  • Host
  • ServiceDescription
  • Severity
  • HostStatus
  • ServiceStatus
  • TextMessage
  • ReportDate
  • LastInsertDate
  • FirstInsertDate
  • SubComponent
  • ErrorType
Event - Host or Service Notification
  • Host
  • ServiceDescription
  • Severity
  • HostStatus
  • ServiceStatus
  • TextMessage
  • ReportDate
  • LastInsertDate
  • FirstInsertDate
  • SubComponent
  • ErrorType
  • LoggerName
Host Groups
  • Name
  • Description
Device
  • DisplayName
  • Description
  • Identification
Monitoring Server
  • MonitorServerName
  • IP
  • Description

6.0 Tutorial: From Theory to Implementation: A Real-World Adapter for SNMP Traps

6.1 Foundation Adapter Development Environment

The following text describes step by step how to develop a new Feeder and Adapter for inserting custom data streams into the Foundation framework. The input source will be SNMP traps. The Adapters are distributed as Java libraries (jar files) and therefore you need the JAVA SDK available for compiling and packaging the Adapters. The tutorial uses ANT and MAVEN as build tools. You can get ANT and Maven binaries from the Apache site. The tutorial uses the Foundation source distribution which is available from http://sourceforge.net/projects/gwfoundation.

A. Before You Start
  1. Install and configure the build tools and Foundation.
  2. Make sure that foundations build runs without errors.
  3. Define the data that needs to be integrated.
  4. What is the Application scope? SNMP trap are treated as a separate application and therefore we have to create a new ApplicationType: SNMPTRAP.
  5. What is the Entity Type? SNMP trap messages will be stored in the LogMessage table and therefore the EntityType is LOG_MESSAGE.
  6. What fields need to be stored? Define the attributes that are generated by the application and specify the if required and the default values if not required.
    || Attribute || Type || Property || Required / Default Value ||
    Host String No Required
    Severity Class Severity No Required
    IpAddress String Yes Required
    MonitorStatus Class Severity No Same as Severity
    ReportDate Date No Default set at time inserted
    LastInsertDate Date No Default set at time inserted. Can be set by SNMP_LOG message
    Event_OID_numeric String Yes Not required, default to NULL
    Event_OID_symbolic String Yes Not required, default to NULL
    Event_Name String Yes
    Not required, default to NULL
    Category String Yes Not required, default to NULL
    Variable_Bindings String Yes Not required, default to NULL
    TextMessage String No Not required, default to NULL
  7. Select fields that are application specific. A list of fields needs to be defined that are properties attached to the table defined as the Entity Type. The properties are application specific and are not part of the base table. For SNMP traps the following fields and types will be stored.
    || Property/Name || Type ||
    IpAddress String
    Event_OID_numeric String
    Event_OID_symbolic String
    Event_Name String
    Category String
    Variable_Bindings String
  8. Select fields for data consolidation. This feature reduces the number of identical messages in the LogMessage table. For each insert, the consolidation criteria will be applied to the incoming message as long as the flag in the XML stream is set to a consolidation criteria name (consolidate="SNMPTRAP" ). By default no consolidation criteria is applied. If the consolidation criteria matches that of an existing message in the Log Message table, the message counter for the existing message will increased and the date fields will be updated as following:
    || Field || Change ||
    FirstInsertDate Unchanged
    LastInsertDate ReportDate
    ReportDate System (current time)

Fields that Need to Match before a Message Gets Consolidated:

  • OperationStatus
  • Host
  • Severity
  • IPaddress
  • MonitorStatus
  • Event_OID_numeric
  • Event_Name
  • Category
  • Variable_Bindings

Once the data set is defined and properties and consolidation criteria are defined, the next step will be to update the database Metadata and write an Adapter for SNMP data normalization.

B. Database Updates

The following database updates can be integrated into one database script that ships with the new Adapter. It's always a good idea to check for the existence of an entry before attempting to insert that entry into the database, and abort if they are already present. But for readability of the tutorial these steps are not documented. Since a new application was created the Application needs to be added to the database; INSERT INTO ApplicationType(Name, Description) VALUES ("SNMPTRAP", "SNMP Trap application").

The new properties need to be created and assigned to the ApplicationType and EntityType:

INSERT INTO PropertyType(Name, Description, isString) VALUES ("ipaddress", "ipdddress of snmp device", 1);
INSERT INTO PropertyType(Name, Description, isString) VALUES ("Event_OID_numeric", "Event_OID_numeric", 1);

Continue for all property types.

INSERT INTO  ApplicationEntityProperty(ApplicationTypeID, EntityTypeID,  PropertyTypeID, SortOrder) VALUES ((SELECT ApplicationTypeID FROM  ApplicationType WHERE Name='SNMPTRAP'),(SELECT EntityTypeID FROM  EntityType WHERE Name='LOG_MESSAGE'),(SELECT PropertyTypeID FROM  PropertyType WHERE Name = 'ipaddress'), 1);
INSERT INTO  ApplicationEntityProperty(ApplicationTypeID, EntityTypeID,  PropertyTypeID, SortOrder) VALUES ((SELECT ApplicationTypeID FROM  ApplicationType WHERE Name='SNMPTRAP'),(SELECT EntityTypeID FROM  EntityType WHERE Name='LOG_MESSAGE'),(SELECT PropertyTypeID FROM  PropertyType WHERE Name = 'Event_OID_numeric'), 1);

Continue for all properties.

The consolidation criteria needs to be named and the criteria of matching fields is a semicolon separated list.

INSERT INTO ConsolidationCriteria(Name, Criteria)  VALUES ("SNMPTRAP", "OperationStatus;Host;Severity;ipaddress,  MonitorStatus;  Event_OID_numeric;Event_Name;Category;Variable_Bindings");
C. Writing the Feeder

The Feeder captures SNMP trap and sends XML formatted messages where all the fields to monitor are XML attributes to the Foundation listener component. The Listener listens on a configurable socket for incoming messages. The default is set to port 4913. For SNMP traps the XML messages look like the following:

<SNMPTRAP
MonitorServerName="localhost"
Host="cisco2900.itgroundwork.com"
Severity="Normal"
MonitorStatus="Normal"
ReportDate=""2005-10-25 04:20.44"
LastInsertDate="2005-10-25 04:20.44"
ipaddress="192.168.2.203"
Event_OID_numeric=".1.3.6.1.4.1.9.0.1"
Event_OID_symbolic="enterprises.9.0.1"
Event_Name="tcpConnectionClose"
Category="Status Events"
Variable_Bindings="enterprises.9.2.9.3.1.1.1.1:5  tcpConnState.192.168.2.203.23.192.168.2.249.38591:synReceived  enterprises.9.2.6.1.1.5.192.168.2.203.23.192.168.2.249.38591:600  enterprises.9.2.6.1.1.1.192.168.2.203.23.192.168.2.249.38591:70  enterprises.9.2.6.1.1.2.192.168.2.203.23.192.168.2.249.38591:101  enterprises.9.2.9.2.1.18.1:"
TextMessage="A tty trap signifies that a TCP connection, previously  established with the sending protocol entity for the purposes of a tty  session, has been terminated. 5 synReceived 600 70 101 " />
D. Writing an Adapter
Creating the Java Project
  1. Inside the expanded Foundation package, go into collagefeeder/adapter and create a new folder called snmp.
  2. Step inside the snmp directory and create a source/java directory.
  3. Create a new maven.xml file that looks like the following:
    <project default="jar:install"
        xmlns:j="jelly:core"
        xmlns:maven="jelly:maven"
        xmlns:ant="jelly:ant">
            <goal name='distro'>
                <attainGoal name='clean'/>
                <attainGoal name='jar'/>
                <delete dir='./dist' />
                <mkdir dir='./dist' />
                <mkdir dir='./dist/lib' />
                <copy todir="./dist/lib" file="${maven.build.dir}/${maven.final.name}.jar"/>
                <j:forEach var="lib" items="${pom.artifacts}">
                    <j:set var="dep" value="${lib.dependency}"/>
                    <j:if test="${dep.getProperty('war.bundle')=='true'}">
                    <copy todir="./dist/lib" file="${lib.path}"/>
                    </j:if>
                </j:forEach>
            </goal>
            <goal name='allBuild'>
                <attainGoal name='distro'/>
            </goal>
    </project>
  4. Create a project.xml that looks like the following. Note that the version and the libraries are inherited from the main project file:
    <project>
        <pomVersion>3</pomVersion>
        <extend>../project.xml</extend>
        <id>collage-adapter-snmp</id>
        <name>Groundwork Collage Adapters for SNMP</name>
        <package>com.groundwork.feeder</package>
            <build>
            <sourceDirectory>src/java</sourceDirectory>
              <resources>
                <resource>
                  <directory>${basedir}/src/java</directory>
                  <excludes>
                    <exclude>**/*.java</exclude>
                  </excludes>
                </resource>
              </resources>
            </build>
    </project>
  5. Create a new package by creating the following directories under src/java: com.groundwork.feeder.adapter.impl.
    Now the setup of the new java project that will include the SNMP Adapter is done. The next step will show what classes need to be implemented.
Classes and Method to Overwrite

The Adapter has to implement the FeederBase interface which is part of the adapter-api. Class creation:

  1. Create a new class SNMPTrap inside com/groundwork/feeder/adapter/impl that implements the FeederBase (Note: will be renamed to AdapterBase).
  2. Implement GetName that returns the Adapter name. The name has to match the node name of the XML fragment sent to the listener. For snmp traps, it is SNMPTRAP.
  3. Implement initialize() and uninitialize() for any actions that need to be executed when the Adapter gets loaded or unloaded by the framework. Implement the method process() that gets called by the framework for each incoming XML stream that is of the Adapter name (SNMPTRAP). Into this method goes the normalization code that transforms the XML message to a database call. The main steps are:
    • Parse the XML stream and extract the attributes. Use the utils.getAttributes() method.
    • Get the API object by calling into the bean factory.
    • Create a properties map and call into the API.
Spring Assemblies for SNMPTrap Adapter

The new SNMPTrap Adapter will be loaded into the Spring container. What you have to include into your JAR file is the spring assembly file which has to be created in the src/java/META-INF directory. Steps:

  1. Create a new folder META-INF inside your project's snmp/src/java directory
  2. In META-INF create a file called assembly-adapter-snmptrap.xml. A sample of the whole file can be seen below under Spring Assembly for SNMPTRAP Adapter.
Building the Adapter Package

Inside the adapter/snmp directory execute: maven jar. This will create the jar file in the target directory. Executing: maven jar:install This will copy the jar file into the local repository.

E. Installing and Configuring the SNMPTRAP Adapter

Once the Adapter is compiled successfully it needs to be deployed into the listener installation and the adapter.properties need to be updated with the new Adapter entry.

  1. Copy the jar file collage-adapter-snmp-1.1.jar into the listener library path /usr/local/groundwork/collage/feeder/lib. Edit adapter.properties for the new adapter as following:
    First, increment the counter for assemblies:
    # Spring assemblies
    nb.assemblies = 3

    Add the assembly name and the Property Bean name:

    # SNMPTrap Beans
    adapter.assembly3 = META-INF/assembly-adapter-snmptrap.xml
    adapter.properties.assembly3 = SNMPTrapAdapterProperties
  2. Start the listener.
  3. Feed data to the listener and verify that the data shows up in the database. Check the log files for any errors.
F. SQL Script for SNMPTRAP Metadata Creation
# Database changes for SNMPTRAP messages
# Add new ApplicationType for SNMPTRAP
DELETE FROM ApplicationEntityProperty WHERE  ApplicationTypeID = (SELECT ApplicationTypeID FROM ApplicationType WHERE  Name='SNMPTRAP') %% EntityTypeID = (SELECT EntityTypeID FROM EntityType  WHERE Name='LOG_MESSAGE');
INSERT INTO ApplicationType (Name, Description) VALUES ("SNMPTRAP","SNMP Trap application");
# Add the properties specific to SNMPTRAP
INSERT INTO PropertyType(Name, Description, isString) VALUES ("IpAddress", "ipadddress of snmp device", 1);

INSERT INTO PropertyType(Name, Description, isString) VALUES ("Event_OID_numeric", "Event_OID_numeric", 1);

INSERT INTO PropertyType(Name, Description, isString) VALUES ("Event_OID_symbolic", "Event_OID_symbolic of snmp device", 1);

INSERT INTO PropertyType(Name, Description, isString) VALUES ("Event_Name", "Event_Name", 1);

INSERT INTO PropertyType(Name, Description, isString) VALUES ("Category", "Category of snmp device", 1);

INSERT INTO PropertyType(Name, Description, isString) VALUES ("Variable_Bindings", "Variable_Bindings", 1);
# Assign the SNMP properties to Application Type SNMPTRAP and Entity LOG_MESSAGE
INSERT INTO ApplicationEntityProperty(ApplicationTypeID, EntityTypeID,  PropertyTypeID, SortOrder) VALUES ((SELECT ApplicationTypeID FROM  ApplicationType WHERE Name='SNMPTRAP'),(SELECT EntityTypeID FROM  EntityType WHERE Name='LOG_MESSAGE'), (SELECT PropertyTypeID FROM  PropertyType WHERE Name = 'IpAddress'), 1);

INSERT INTO ApplicationEntityProperty(ApplicationTypeID, EntityTypeID,  PropertyTypeID, SortOrder) VALUES ((SELECT ApplicationTypeID FROM  ApplicationType WHERE Name='SNMPTRAP'),(SELECT EntityTypeID FROM  EntityType WHERE Name='LOG_MESSAGE'), (SELECT PropertyTypeID FROM  PropertyType WHERE Name = 'Event_OID_numeric'), 1);

INSERT INTO ApplicationEntityProperty(ApplicationTypeID, EntityTypeID, PropertyTypeID, SortOrder) VALUES ((SELECT ApplicationTypeID FROM  ApplicationType WHERE Name='SNMPTRAP'),(SELECT EntityTypeID FROM  EntityType WHERE Name='LOG_MESSAGE'),(SELECT PropertyTypeID FROM  PropertyType WHERE Name = 'Event_OID_symbolic'), 1);

INSERT INTO ApplicationEntityProperty(ApplicationTypeID, EntityTypeID,  PropertyTypeID, SortOrder) VALUES ((SELECT ApplicationTypeID FROM  ApplicationType WHERE Name='SNMPTRAP'), (SELECT EntityTypeID FROM  EntityType WHERE Name='LOG_MESSAGE'),(SELECT PropertyTypeID FROM  PropertyType WHERE Name = 'Event_Name'), 1);

INSERT INTO ApplicationEntityProperty(ApplicationTypeID, EntityTypeID, PropertyTypeID, SortOrder) VALUES ((SELECT ApplicationTypeID FROM  ApplicationType WHERE Name='SNMPTRAP'), (SELECT EntityTypeID FROM  EntityType WHERE Name='LOG_MESSAGE'), (SELECT PropertyTypeID FROM  PropertyType WHERE Name = 'Category'), 1);

INSERT INTO ApplicationEntityProperty(ApplicationTypeID, EntityTypeID,  PropertyTypeID, SortOrder) VALUES ((SELECT ApplicationTypeID FROM  ApplicationType WHERE Name='SNMPTRAP'),(SELECT EntityTypeID FROM  EntityType WHERE Name='LOG_MESSAGE'),(SELECT PropertyTypeID FROM  PropertyType WHERE Name = 'Variable_Bindings'), 1);
#Create consolidation criteria
INSERT INTO ConsolidationCriteria(Name, Criteria)  VALUES ('SNMPTRAP',  'Host;Severity;IpAddress;MonitorStatus;Event_OID_numeric;Event_Name;Category;Variable_Bindings')
G. Spring Assembly for SNMPTrap Adapter

The following file assembly-adapter-snmptrap.xml needs to be included into the jar package for the SNMPTrap Adapter inside META-INF:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">
<beans>
<!--
List all the BeanID that have implemented the initialize method. The bean ID's defined as a comma
separated list will be called during the loading of the assembly
-->
<bean id="SNMPTrapAdapterProperties" class="com.groundwork.feeder.adapter.impl.AdapterProperties">
<constructor-arg type="java.lang.String"><value>adapter.snmptrap</value></constructor-arg>
 </bean>
<bean id="adapter.snmptrap" singleton="false"
      class="com.groundwork.feeder.adapter.impl.SNMPTrap" />
</beans>
H. Creating a Feeder for LOG4J and Using the Generic Log Adapter

The Foundation Adapters are pluggable modules, written in Java, that normalize data from an Application before it gets inserted into the data store. Normalization is needed for complex data structures, but a simple Generic Adapter with several pre-defined fields might be sufficient for many external log messages, including some application logs. Foundation provides a GenericLog Adapter that can be used to add log data into Foundation without writing any code. The values assigned to pre-defined fields will be stored in the database under a user defined Application Type. The following example shows how LOG4J (used by Application Servers) can be fed into Foundation.

Installing and Using the Generic Log Adapter

The Generic Log Adapter is included in all distributions of Foundation.

Definitions

An ApplicationType for the Adapter to log data against must be defined. In this example the ApplicationType will be LOG4J, but this is user defined and may be adjusted to any type.

Generic Adapter Input

The following attributes are required as part of the XML feed, otherwise the message will be rejected: ApplicationType, MonitorServerName, Device, Severity, TextMessage

In addition to the required properties the following properties are accepted but optional: Host, MonitorStatus, ReportDate, OperationStatus, ApplicationSeverity, Component, Priority, ServiceDescription, Priority, and TypeRule.

Notes about some fields:

  • If OperationStatus is not defined it will be set to Open.
  • If ReportDate is not set the current time (system) will be used.
  • If MonitorStatus is not defined it will be set to UNKNOWN. MonitorStatus defines the color in the console: OK = Green, DOWN = Red, WARNING = Yellow, and UNKNOWN = Blue.
  • If the FeederScript defines values for the Host and ServiceDescription attributes, the LogMessage will be linked (via a Foreign Key) to the matching ServiceStatus. This can be used in the GroundWork Monitor UI to link from an Event Message to the ServiceStatus.
  • If Host is set the LogMessage will be linked to HostStatus.
  • If Host and Service Description are set the message will be linked to an existing ServiceStatus entry.
  • If the FeederScript defines a value for the Host attribute, the LogMessage will be linked (via a Foreign Key) to the matchng HostStatus entry.
System Setup

Create a new ApplicationType (LOG4J) in the database:

Stop the listener and update the database with a SQL statement. In GroundWork Monitor, these statements will work:

Stop gwservice from the bash command line:

# /usr/local/groundwork/ctlscript.sh stop gwservices

PostgreSQL command line:

# psql -U postgres gwcollagedb
gwcollagedb=# INSERT INTO ApplicationType (Name, Description) VALUES("LOG4J","LOG4J Events");
gwcollagedb=#\q

Start GWService:

# /usr/local/groundwork/ctlscript.sh start gwservices
I. Feeder Code Example

Here is an example of the simplest possible script that can be used to send one message to the Event table:

#!/usr/local/groundwork/perl/bin/perl --
#
#Copyright 2003-2011 Groundwork, Inc.
#[http://www.gwos.com]
#
#Unless required by applicable law or agreed to in writing, software
#distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
#WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
#License for the specific language governing permissions and limitations under the License.
#
use IO::Socket;
my $debug =1 ;
my $remote_host = "localhost";
my $remote_port = 4913;
my $socket;
if ( $socket = IO::Socket::INET->new(PeerAddr => $remote_host,
PeerPort => $remote_port,
Proto    => "tcp",
Type     => SOCK_STREAM)
) {
my $xml_message = "<GENERICLOG ApplicationType='LOG4J' MonitorServerName='localhost' Host='dashboard.itgroundwork.com' Device='dashboard.itgroundwork.com' Severity='WARNING' MonitorStatus='WARNING' ErrorType='LogRotation' SubComponent='LOG4J Integrator' TextMessage='16:15:54,593 [WARN ] com.groundwork.collage.impl.LogMessageDAOImpl - Consolidation criteria matches with more than one record. Make sure the criteria is better defined. If the consolidation criteria was turned on after identical messages were inserted you have to run consolidate existing messages. Contact support for more information about database maintenance.' />" ;
              print $xml_message."\n\n" if $debug;
              print $socket $xml_message;
              my $xml_message = "<SERVICE-MAINTENANCE command=\"close\" />";
              print $xml_message."\n\n" if $debug;
              print $socket $xml_message;
             } else {
print "Couldn't connect to $remote_host:$remote_port : $@\n";
}
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.