Foundation Adapter Development - The following text describes step by step how to develop a new Feeder and Adapter for inserting custom data streams into the Foundation framework. The input source will be SNMP Traps. Development Environment - The Adapters are distributed as Java libraries (jar files) and therefore you need the JAVA SDK available for compiling and packaging the Adapters. The tutorial uses ANT and MAVEN as build tools. You can get ANT and Maven binaries from the Apache site. The tutorial uses the Foundation source distribution which is available from http://sourceforge.net/projects/gwfoundation.
- Install and configure the build tools and Foundation.
- Make sure that foundations build runs without errors.
- Define the data that needs to be integrated.
- What is the Application scope? SNMP traps are treated as a separate application and therefore we have to create a new ApplicationType: SNMPTRAP.
- What is the Entity Type? SNMP trap messages will be stored in the LogMessage table and therefore the EntityType is LOG_MESSAGE.
- What fields need to be stored? Define the attributes that are generated by the application and specify the if required and the default values if not required.
Table: Fields to be stored
Attribute Type Property Required/Default Value Host String No Required Severity Class Severity No Required IpAddress String Yes Required MonitorStatus Class Severity No Same as Severity ReportDate Date No Default set at time inserted LastInsertDate Date No Default set at time inserted. Can be set by SNMP_LOG message Event_OID_numeric String Yes Not required, default to NULL Event_OID_symbolic String Yes Not required, default to NULL Event_Name String Yes
Not required, default to NULL Category String Yes Not required, default to NULL Variable_Bindings String Yes Not required, default to NULL TextMessage String No Not required, default to NULL
- Select fields that are application specific. A list of fields needs to be defined that are properties attached to the table defined as the Entity Type. The properties are application specific and are not part of the base table. For SNMP traps the following fields and types will be stored.
Table: Defined fields
Property/Name Type IpAddress String Event_OID_numeric String Event_OID_symbolic String Event_Name String Category String Variable_Bindings String
- Select fields for data consolidation. This feature reduces the number of identical messages in the LogMessage table. For each insert, the consolidation criteria will be applied to the incoming message as long as the flag in the XML stream is set to a consolidation criteria name (consolidate="SNMPTRAP" ). By default no consolidation criteria is applied. If the consolidation criteria matches that of an existing message in the Log Message table, the message counter for the existing message will increased and the date fields will be updated as following:
Table: Fields for data consolidation
Field Change FirstInsertDate Unchanged LastInsertDate ReportDate ReportDate System (current time)
Fields that Need to Match before a Message Gets Consolidated:
Once the data set is defined and properties and consolidation criteria are defined, the next step will be to update the database Metadata and write an Adapter for SNMP data normalization.
The following database updates can be integrated into one database script that ships with the new Adapter. It's always a good idea to check for the existence of an entry before attempting to insert that entry into the database, and abort if they are already present. But for readability of the tutorial these steps are not documented. Since a new application was created the Application needs to be added to the database; INSERT INTO ApplicationType(Name, Description) VALUES ("SNMPTRAP", "SNMP Trap application").
The new properties need to be created and assigned to the ApplicationType and EntityType:
Continue for all property types.
Continue for all properties.
The consolidation criteria needs to be named and the criteria of matching fields is a semicolon separated list.
The Feeder captures SNMP trap and sends XML formatted messages where all the fields to monitor are XML attributes to the Foundation listener component. The Listener listens on a configurable socket for incoming messages. The default is set to port 4913. For SNMP traps the XML messages look like the following:
- Inside the expanded Foundation package, go into collagefeeder/adapter and create a new folder called snmp.
- Step inside the snmp directory and create a source/java directory.
- Create a new maven.xml file that looks like the following:
- Create a project.xml that looks like the following. Note that the version and the libraries are inherited from the main project file:
- Create a new package by creating the following directories under src/java: com.groundwork.feeder.adapter.impl.
Now the setup of the new java project that will include the SNMP Adapter is done. The next step will show what classes need to be implemented.
The Adapter has to implement the FeederBase interface which is part of the adapter-api. Class creation:
- Create a new class SNMPTrap inside com/groundwork/feeder/adapter/impl that implements the FeederBase (Note: will be renamed to AdapterBase).
- Implement GetName that returns the Adapter name. The name has to match the node name of the XML fragment sent to the listener. For snmp traps, it is SNMPTRAP.
- Implement initialize() and uninitialize() for any actions that need to be executed when the Adapter gets loaded or unloaded by the framework. Implement the method process() that gets called by the framework for each incoming XML stream that is of the Adapter name (SNMPTRAP). Into this method goes the normalization code that transforms the XML message to a database call. The main steps are:
- Parse the XML stream and extract the attributes. Use the utils.getAttributes() method.
- Get the API object by calling into the bean factory.
- Create a properties map and call into the API.
The new SNMPTrap Adapter will be loaded into the Spring container. What you have to include into your JAR file is the spring assembly file which has to be created in the src/java/META-INF directory. Steps:
- Create a new folder META-INF inside your project's snmp/src/java directory
- In META-INF create a file called assembly-adapter-snmptrap.xml. A sample of the whole file can be seen below under Spring Assembly for SNMPTRAP Adapter.
Inside the adapter/snmp directory execute: maven jar. This will create the jar file in the target directory. Executing: maven jar:install This will copy the jar file into the local repository.
Once the Adapter is compiled successfully it needs to be deployed into the listener installation and the adapter.properties need to be updated with the new Adapter entry.
- Copy the jar file collage-adapter-snmp-1.1.jar into the listener library path /usr/local/groundwork/collage/feeder/lib. Edit adapter.properties for the new adapter as following:
First, increment the counter for assemblies:
Add the assembly name and the Property Bean name:
- Start the listener.
- Feed data to the listener and verify that the data shows up in the database. Check the log files for any errors.
The following file assembly-adapter-snmptrap.xml needs to be included into the jar package for the SNMPTrap Adapter inside META-INF:
The Foundation Adapters are pluggable modules, written in Java, that normalize data from an Application before it gets inserted into the data store. Normalization is needed for complex data structures, but a simple Generic Adapter with several pre-defined fields might be sufficient for many external log messages, including some application logs. Foundation provides a GenericLog Adapter that can be used to add log data into Foundation without writing any code. The values assigned to pre-defined fields will be stored in the database under a user defined Application Type. The following example shows how LOG4J (used by Application Servers) can be fed into Foundation.
The Generic Log Adapter is included in all distributions of Foundation.
An ApplicationType for the Adapter to log data against must be defined. In this example the ApplicationType will be LOG4J, but this is user defined and may be adjusted to any type.
The following attributes are required as part of the XML feed, otherwise the message will be rejected: ApplicationType, MonitorServerName, Device, Severity, TextMessage
In addition to the required properties the following properties are accepted but optional: Host, MonitorStatus, ReportDate, OperationStatus, ApplicationSeverity, Component, Priority, ServiceDescription, Priority, and TypeRule.
Notes about some fields:
- If OperationStatus is not defined it will be set to Open.
- If ReportDate is not set the current time (system) will be used.
- If MonitorStatus is not defined it will be set to UNKNOWN. MonitorStatus defines the color in the console: OK = Green, DOWN = Red, WARNING = Yellow, and UNKNOWN = Blue.
- If the FeederScript defines values for the Host and ServiceDescription attributes, the LogMessage will be linked (via a Foreign Key) to the matching ServiceStatus. This can be used in the GroundWork Monitor UI to link from an Event Message to the ServiceStatus.
- If Host is set the LogMessage will be linked to HostStatus.
- If Host and Service Description are set the message will be linked to an existing ServiceStatus entry.
- If the FeederScript defines a value for the Host attribute, the LogMessage will be linked (via a Foreign Key) to the matchng HostStatus entry.
Create a new ApplicationType (LOG4J) in the database:
Stop the listener and update the database with a SQL statement. In GroundWork Monitor, these statements will work:
Stop gwservices from the bash command line:
PostgreSQL command line:
Here is an example of the simplest possible script that can be used to send one message to the Event table: