This section reviews GroundWork Monitor Foundation.
WAS THIS PAGE HELPFUL?
Foundation is an IT management data abstraction layer and development platform. The Foundation data model allows the integration of any state, event, and performance data, independent of the Monitoring Application that produces it. This offers the possibility to store data for additional systems, including open source and commercial monitoring systems, databases, and even hardware, such as detectors or sensors. It also allows the integration of Application Monitoring data known as MBeans.
The intent of Foundation is to provide a data model that integrates the components of an IT infrastructure requiring monitoring. Flexible methods of integrating data into the data store are provided allowing different tools and applications and databases to feed data into Foundation. Foundation will normalize the data so it can be retrieved in a consistent manner. Foundation then provides various APIs to allow the normalized data to be retrieved. The Foundation package includes Nagios, as the main monitoring system, integrated with Foundation and a set of applications which use the Foundation APIs to present real-time views and reports.
The Web Service interface is a new addition to the existing Foundation Framework. The previous API components have been re-packaged to use the Web Service interface and have not been replaced.
APIs for Perl, PHP, and Web Service are part of the toolkit. The Perl API is a module called CollageQuery that allows a Perl program to retrieve data from Collage. The GroundWork PHP API provides an object library to access the Foundation database sources and the information within these sources. The library is split up into connection and query type classes. Adding a Web Service layer enables more applications to use and integrate with the existing Foundation framework. We'll review adding a Web Service layer which enables more applications to use and integrate with the existing Foundation framework. And, the Foundation 1.5 consolidation feature allows you to reduce the number of LogMessages by creating just one entry for messages that are alike, incrementing the Message counter, and adjusting the date fields. The actual implementation of the APIs is part of the GroundWork Monitor installation.
Programmers can use PHP, Perl, or Web Service API or third party products to:
- Build custom displays of real-time monitoring information
- Build custom reports from historical monitoring information
- Monitor additional devices or systems by feeding monitoring information into the system
- Consolidate information from disparate systems into a single view
- Integrate heterogeneous systems by using GroundWork Foundation as an intermediary system
Foundation is packaged and deployed as a web application (.war) into the Jetty Servlet container. Jetty is an open-source, standards-based, full-featured lightweight servlet container implemented entirely in Java. In addition to the Foundation application the GroundWork implementation includes the following web applications:
- GroundWork Report Server and Eclipse BIRT Viewer that allow to run and manage Eclipse BIRT Reports created with Eclipse BIRT Report Designer.
- GroundWork JMS. A full featured persistence and topic server based on the Open Source project JORAM.
- GroundWork Monitor includes Event Console implemented as a Web application.
The Monarch tool is a web application that is used to configure the Nagios system. It stores the Nagios configuration data in its own database. At this point in time, the Monarch database is separate from the Foundation database. On a monarch commit the new configuration is synchronized with Foundation.
Since the Nagios monitoring system is integrated in this package, any information gathered by a Nagios plugin can be integrated into the system. Nagios Feeders or the Event Broker take the information from the Nagios system and inserts it into the Foundation database. The data objects contained in Foundation map closely to the Nagios objects listed below.
APIs built on top of the Foundation framework allow this information to be retrieved. The APIs allow programs to query by object and data type. Separate APIs are available for Java, PHP, and Perl programs. In addition to the provided samples, the Foundation status views (Overview and Troubleview) are built using the PHP Foundation API.
- Host Groups - This includes Hosts as members.
- Hosts - This typically represents a monitoring entity that in Nagios usually maps to physical devices. A Host entry contains one or more Service Checks.
- Services - This typically represents a Nagios Service Check for a specific Host. A Host-Service combination is unique in the monitoring instance.
The following type of information can be retrieved:
- Host Status - This represents the current status and attributes of Host objects.
- Service Status - This represents the current status and attributes of Service objects.
- Events - These are typically timestamped messages that are generated by a monitoring system or managed device. The following Nagios Events are stored in the LogMessage table.
- Host Alerts - Events generated when a Host changes state.
- Host Notifications - Events generated when a notification occurs based on a Host Alert event.
- Service Alerts - Events generated when a service changes state.
- Service Notifications - Events generated when a notification occurs based on a Service Alert event.
- Nagios Acknowledge - Existing Host or Service Alerts are updated when a user acknowledges messages.
The Foundation framework consists of five main components. Select the diagram icon above to view the Foundation Components:
- Feeders These are scripts or programs which generate a data set that is sent to the Foundation Listener. The protocol is a simple XML stream.
- Foundation Listener This is a port or Java Message Service (JMS) Listener which receives the XML streams from various Feeders and dispatches them to data normalizers, called Adapters.
- Foundation Adapters These Adapters are programs within the Foundation framework that apply rules and data normalization to incoming data. Each Adapter is application specific (e.g. NagiosEvent, SNMP or Syslog) and is easily added and managed with the framework.
- Foundation Persistence Service This is a Relational Persistence layer which runs on the top of a database or a database cluster.
- Foundation API This is documented API's for PHP and Perl, used to retrieve data from the data store.
Figure: Foundation architecture
The data flow of messages is unidirectional, since the Foundation Framework doesn't reply to the incoming XML streams.
Figure: Foundation data flow
In order to integrate data into the Foundation framework, the data generated by the source application (e.g. Nagios or Java Management Extensions (JMX) Service) needs to be read and sent as an XML stream to one of the listeners. This functionality is provided by Feeders. A Feeder can be written in any language; for example, Nagios Feeders are written in Perl. The XML output protocol is simple:
The FeederName matches with the Adapter name, and the Attributes are just a list of name value pairs. For example, the Nagios Event Feeder XML has the following format:
The Feeder could include the logic for normalizing the data, but this is discouraged. The best approach is to have a simple and generic Feeder that reads and forwards the data to the Listener. Normalization functions are best performed by Adapters.
The simple format of an XML element represents as well one transaction across the system. For a large load this is expensive and affects the overall message throughput since transaction carry some overhead. The recent version of Foundation includes support for more complex messages where multiple messages can be bundled into one transaction. More details about the different adapters can be found in the "Data Integration approaches section".
- Foundation Listener
The Listener is a simple service, either listening on port 4913 or on a JMS topic. The incoming XML message is analyzed and forwarded to the appropriate Adapter as defined in the XML element (e.g., the Adapter that matches FeederName).
Adapters are data normalizers that apply normalization or simplification rules to the incoming XML message. For example, an Adapter could calculate the average temperature for a data feed of 10 sensors in a server room, and insert the calculated value into the data store.
An Adapter can be used to validate incoming data for completeness. It can and should be used to reject incomplete or faulty data before it gets rejected by the persistence layer, which would affect system performance.
Adapters are written in Java and compiled into a jar library package. The package includes a Spring assembly file which is read by the Foundation Framework at initialization time. See the tutorial link above for more details about the syntax of the assembly file, and how to deploy an Adapter.
Before you start integrating data into the Foundation data store, you need to decide the following:
- How to collect data from the source application and how to write the Feeder.
- Where the data normalization takes place (Feeder or Adapter).
- Whether the default fields in the data model are enough to store your data, or whether you need to add application specific properties.
- What ApplicationType to use for your data. The ApplicationType is a parameter that allows you to access your data using a simple filter. In GroundWork Monitor, this filter is built in to the Console application view, and will show up automatically when data is present with the application type in question.
- Does it make sense to bundle messages into a single transaction. Bundling would allow a higher message throughput and submit all or nothing of depended data. It adds more complexity to the feeder creating feeds. Foundation added support for this type of messages by defining an XML Schema (link to file) and a new Adapter called SystemAdapter. The recent version of Foundation uses this approach for processing Nagios Status and Event messages.
Foundation comes with a set of Adapters for different type of data feeds. The adapters can be classified into two different types:
- Single Transaction messages
The adapters of this type accept XML feed of the format:
Each XML element represents a transaction. Foundation supports the following adapters.
Adapter Type XML Comments NAGIOS Events <NAGIOS_LOG attribute,.. /> Events from Nagios Nagios Status <SERVICE_STATUS attributes,./>
Host and Service status updates SNMP Trap events <SNMPTRAP attribute,.. /> SNMP trap events coming from the SNMPTT daemon SNMP Trap events <SNMPTRAP attribute,.. /> SNMP trap events coming from the SNMPTT daemon Syslog events <SYSLOG attribute,.. /> Syslog messages from the gw_syslog plugin Generic Events <GENERIC_LOG attribute,../> Generic adapter that maps attributes directly to dynamic properties without checking. System messages <COLLAGE_LOG attribute,../> Reporting System messages to Console
- User definable transaction messages
The preferred way to feed data is using the new SystemAdapter which allows the sending of multiple messages of different entities (Event or Status) in a single transaction. The advantages of this approach are:
- Higher message throughput under load.
- The SystemAdapter uses an XML schema to validate the feed upfront. Any synatx errors are detected before being processed in the service layer.
View XML Schema: SystemConfig.pdf
- Wrapping dependent messages in a transaction. If any of the message fail everything will roll back and guarantee data consistency.
For information on how to use and configure feeds that use the SystemAdmin adapter see example in the section Configuring Data Feeders in this document.
As mentioned in the earlier chapters, the Feeder has to produce an XML stream that can be sent to one of Foundation's Listener services. If the Feeder performs normalization, or if the input data is simple and matches the default data model properties, then the Feeder can send the data to the Generic Adapter. The Generic Adapter maps the sent attributes to database properties without validating the values.
A sample of how to feed data to the Generic Adapter can be found below under Creating a Feeder for LOG4J and Using the Generic Log Adapter.
If you decide that your application needs more properties to be stored along with the default Status and Event data fields, the following steps are necessary:
- Add new properties and their type to the PropertyType table.
- Associate the properties with the EntityType such as LOG_MESSAGE, SERVICE_STATUS, or HOST_STATUS.
- Define an ApplicationType for your data.
In the current version of GroundWork Foundation the above database operation needs to be executed with SQL statements. These methods will be supported by the next update of the Admin Feeder, which will allow dynamic addition of properties via the input stream.
Custom properties can be inserted using the Generic Adapter, but since no consistency checking is applied, message feeds with missing properties will be rejected.
- For large numbers of constant data feeds, implementation of an Adapter to validate the incoming data (all required fields available, correct type) is recommended. This will ensure that any error that would cause a transaction rollback (an expensive middle layer operation) can be detected up front and rejected.
- Feeders should be simple and as generic as possible - collect data points and send them to the Adapter for normalization. This reduces the load of concurrently running Feeder processes, which can be inefficient, especially when written in interpreted languages such as Perl.
- Use the SystemAdapter whenever possible. The performance improvements and the improved message validation make the system much more robust.
Foundation is a system of several loosely coupled components described below:
- Foundation-webbapp - A core component that includes the business objects, the data persistent component (Object Relation Mapping ORM), the data normalizer components (Foundation adapters), and the Web Service API (Soap based API).
- Foundation-JMS - The server hosting the Message Queue for incoming data feeds and a Topic server for notification.
- Foundation-reportserver - An application to manage BIRT reports.
- birtviewer - An Eclipse application to view reports generated with Eclipse BIRT Report Designer.
Each of the components are build as a Web Application and deployed into the servlet container (Jetty). In addition to the web applications Foundation includes the following components:
- Nagios Feeders - The Nagios feeders read Nagios status and log files and send XML messages to Foundation. The feeder scripts are located in /usr/local/groundwork/foundation/feeder directory and are named nagios2collage_status.pl which reads the Nagios status log and updates the Status database with Host and Service status information.
- GroundWork Web Service Plugin for Eclipse - The plugin is included in the distribution (usr/local/groundwork/foundation/eclipse) but as well bundled with the BIRT Viewer web component.
- Foundation Files and Components - /usr/local/groundwork/foundation
- Web Applications - foundation/container/webapps
- Context Files for Web Applications - foundation/container/contexts
- Configuration for the Jetty servlet Container - foundation/container/etc
Foundation configuration uses properties files are stored in /usr/local/groundwork/config. Changing any of these files requires a restart of gwservices.
|db.properties||Contains the database credentials for any database used by GroundWork Monitor|
|foundation.properties||Defines runtime configuration such as the port to listen on, location and configuration of the JMS server and properties to tune the application such as size of the different thread and connection pools. See Settings for GroundWork adapters.|
|adapter.properties||List of adapters (Normalizer components) used for message processing.|
|gwreportserver.properties||Defines the location of the reports and information about the report viewer.|
|log4j.xml||This file allows to change the level of log reporting for all Java applications. By default is set to Error only. For debugging purposes it can be set to Warning, info or debug.|
The output of the log files are defined in log4j.xml. By default all the log files for Java go to the directory:
To start foundation issue the following command (needs root privileges):
and similar to stop the service:
|Caution needs to be applied when starting and stopping services since stopping foundation will shutdown the message listeners and the API driving the User Interface screens.|
The following section describes configuring the system for custom data integration and the steps necessary to setup and configure data feeds into GroundWork Monitor for status and event monitoring.
An application, let's call it TemperatureWatcher, monitors 10 temperature sensors in different rooms of a building. At startup and shutdown the TemperatureWatcher applications sends events. In normal operations every 30 seconds the temperature of each sensor is sent to GroundWork Monitor.
Additionally, the TemperatureWatcher checks the status of the sensors. If the status is different from an OK status and the sensors do not respond an event is sent to GroundWork Monitor.
The Events generated by TemperatureWatcher should be visible in the Event Console while the temperature of each sensor should be visible in the Status application.
Given the above specifications the following meta data needs to be generated in Foundation:
- ApplicationType: TempWatcher
- Custom Property for Status information: Temperature of type Double - This is the field where the temperature measurements are stored.
- Host Group: TemperatureWatcherApps - The HostGroup needs to be created so that the TemperatureWatcher applications are visible in Status.
The easiest way to insert the metadata into Foundation is to stop Foundation, update SQL and restart Foundation.
- Stop Foundation from the command line logged in as root:
- Update SQL data into the database:
- Restart Foundation:
Configuration Data for the Application, Checks and Hostgroup are XML feeds to the TCP port 4913 using the SystemAdmin adapter for Data Normalization. The following Feed creates the necessary entries so that the system can accept monitoring data from the TemperatureWatcher application:
Any data (Status and Events) will be sent to the SystemAdmin adapter since the data normalization is done by the feeder application.
Sending an event will use the following XML feed send to the TCP port 4913. The example reports a FATAL error on the sensor_1 check:
For best performance the status of all the 10 sensors should be sent as one message to Foundation:
With this configuration Status Data is visible in Status under the HostGroup TempWatcherApp and all Events are visible in the Event Console.
The Foundation consolidation feature allows you to reduce the number of LogMessages by creating just one entry for messages that are alike, incrementing the Message counter, and adjusting the date fields. Consolidation processing is applied to all incoming messages of Entity Type LogMessage where the attribute consolidation is defined (e.g. consolidation='SNMPTRAP').
Consolidation criteria is stored in the database and consists of a user definable Name and the Criteria. The criteria is a semicolon (;) separated list of LogMessage record fields or PropertyName. For an incoming message to be consolidated, all values of the fields defined in the Criteria have to match an existing record.
An incoming message must define the Name of the consolidation criteria that it will be matched against. If the consolidation tag is missing, a new log entry is created, which is the system's default behavior.
By default, the consolidation is turned on for Nagios, SNMP, and Syslog event processing. Consolidation will occur if the fields defined in the consolidation criteria match. An exception to the above is if the Monitoring Status of the last event is different than the Monitoring Status of the incoming event, a new console message will be created. This rule guarantees that console messages sorted by chronological order will always show the current status above previous status. The following consolidation criteria is defined in Foundation's ConsolidationCriteria table. An Administrator can update consolidation criteria entries to the consolidation criteria table within GroundWork Monitor Administration>Foundation>Manage Consolidation Criteria. Here you will find existing consolidation criteria and a place to enter and add new criteria.
Table: Consolidation Criteria
|snmptt forwarding traps to Foundation|
Figure: Updating consolidation criteria
To disable consolidation for the Feeders shipped with GroundWork Monitor, you can modify the Nagios event log feeder, nagios2collage_eventlog.pl, to send a message to Foundation that does not specify consolidation. Change the script that contains the following line:
To the following:
To enable consolidation, the event message needs to look like the following, followed by the other arguments:
The Consolidation Criteria in the database for NAGIOSEVENT defines the following criteria:
If an event is fed to the system and the consolidation criteria is defined, the system checks if any log message for the Device, MonitorStatus, OperationStatus, SubComponent and ErrorType already exists. The system will only consolidate if:
- just one LogMessage object matches
- if multiple LogMessage objects match a warning is logged indicating that the criteria needs to be better defined
If a match is found an existing message the message counter LogMessage.MsgCount for an existing message will be incremented and the date fields will be updated as follows:
|Database Field||Change Applied|
|ReportDate||System (current time)|
|TextMessage||Updated Text message. Text Message might include the values of a check (85% disk used) that changes while the status and the type remain the same.|
Host Groups, Device, and Monitoring Server attributes are processed by the SystemAdmin adapter.
Foundation's data store has a flexible data format that can be expanded if needed. The Foundation data objects and attributes implemented in the Foundation package that are applicable to a Nagios-centric are listed below.
|The attributes listed below are Nagios specific and apply to the application type Nagios. The Nagios Adpaters (Nagios_log (Events), Host_Status Service_Status) check for the existence of the following attributes.|