Icinga 2

1.0 Prerequisites

1.1 Enable the API

The Icinga 2 system you run needs to have the API feature enabled. By default, it is not in the current version of Icinga 2 (2.5.4). It is installed, however, and it is a relatively simple matter to set it up. Please see the relevant documentation at http://docs.icinga.org.

Assuming you have Icinga 2 installed properly, and the API is not enabled, the commands will guide you through the basics:
```
# icinga2 api setup
# service icinga2 restart
```
The data you will need to connect to the API will be in the file:
```
/etc/icinga2/conf.d/api-users.conf
```

It will look something like this:

object ApiUser "root" {
  password = "3393c21c662f1b42"
  // client_cn = ""
  permissions = [ "*" ]

This defines a user (root) and password for connecting to the API, and technically this is all you really need. You can test with a curl command, like this:
```
# curl -k -s -u root:3393c21c662f1b42 'https://{icinga2 server name here}:5665/v1/status'
```
That should return some XML to the command line with a lot of metrics embedded in it. Note, however, that the data is retrieved over HTTPS, so an SSL certificate has been generated.
If you want to simply trust this certificate, you can choose to do so in the connector, If you wish to import the certificate and not simply blindly trust it (which is a good idea), you will need to do a few more steps.

1.2 Copy and import the certificate

On the Icinga 2 server, suitable responsive to a curl command with username and password over HTTPS as illustrated above, find the ca.crt file and securely copy it to the GroundWork server. It is located here by default on the Icinga 2 server:
```
/etc/icinga2/pki/ca.crt
```

On the GroundWork server, create a new, empty directory in a secure location on the disk, accessible to user nagios. For example, as root, type:

# mkdir /usr/local/groundwork/config/cloudhub/icinga2/
# chown nagios.nagios /usr/local/groundwork/config/cloudhub/icinga2/
# cd /usr/local/groundwork/config/cloudhub/icinga2/

Copy the ca.crt file to that directory, and change directory to that location. Then type issue the following commands as root:

# source /usr/local/groundwork/scripts/setenv.sh
# keytool -genkeypair -alias private -keyalg RSA -keysize 2048 -dname "CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown" -keypass {password one} -keystore icinga2-keystore.jks -storepass {password two}

Replace the password strings in the above command with passwords you maintain for this. Make sure they are unique, and that you record them securely.
```
# keytool -importcert -trustcacerts -noprompt -alias root -file ca.crt -keystore icinga2-keystore.jks -storepass {password two}
```

Then test that the certificate is in place with:

# keytool -list -keystore icinga2-keystore.jks -storepass {password two}

2.0 Managing an Icinga 2 Connection

This section reviews how to add and configure the Cloud Hub connector Icinga 2. Each connector requires a unique set of parameters (e.g. server url, credentials). You will need your GroundWork server and virtual environment connection parameters handy.

2.1 Adding a new connection

Log in to GroundWork Monitor as an Administrator.
Select GroundWork Administration > GroundWork Cloud Hub. The Cloud Hub Configuration Wizard screen will be displayed where you can add and configure the Cloud Hub for various virtual environments. For each of the established configurations you can start or stop the connection, modify the parameters, or choose to remove a connection.
To start a new connection click the +Add icon next to the environment to add. You will create a new connection in this way for each region in Icinga 2 that is to be monitored.

Figure: Cloud Hub Configuration Wizard

2.2 Configuring GroundWork server values

Next, enter the GroundWork server values to access the region. You will need to point the Cloud Hub Icinga 2 connector to a GroundWork server, indicate if it supports SSL, and give it an API key to transmit data.

Figure: GroundWork server values for Icinga 2 (Example)

Display Name: This is the configuration server display name.
GroundWork Server Name: You will need to enter the name of the GroundWork server that will integrate the Cloud Hub messages. If Cloud Hub is running on the same server as the portal the name can be localhost, or as preferred the server name.
Is SSL enabled on GroundWork Server?: Check this box if the GroundWork server is configured for secure HTTPS.

GroundWork Web Services Username and Password: User and password configured to access the Web Services API. These can be obtained by opening a tab to the GroundWork Administration > GroundWork License page. These are the same credentials set within /usr/local/groundwork/config/ws_client.properties.

Important for LDAP enabled systems: Make sure that it matches with the entry in the ws_client.properties file and the user is member of the Authenticated group and the WSUser (or GWUser) group in LDAP.
Without 7.0.2 SP3: The Web Services user name may be different if you are using LDAP and GroundWork Monitor 7.0.2 without the SP3 patch. In this case make an adjustment to what you see in the image below to match what you have, and also fill in the accurate password.
With 7.0.2 SP3: If you applied the SP3 patch the Web Services user will not have a password, instead you need to fill in the token from the GroundWork Administration > GroundWork License page. Under the title Webs Services API Account Info the default encrypted token can be copied into the Cloud Hub page.

Merge hosts on GroundWork Server?: If checked, this option combines all metrics of same named hosts under one host. For example, if there is a Nagios configured host named demo1 and a Cloud Hub discovered host named demo1, the services for both configured and discovered hosts will be combined under the hostname demo1 (case-sensitive).

2.3 Configuring virtualization server values

Next, we continue with the second half of the configuration wizard by entering the values for the virtualization server. The data that the GroundWork server receives comes from the Icinga 2 server, the information is pulled from the API on a periodic basis based on the interval that is set.

Figure: Values for a Icinga 2 connection (Example)
- Icinga 2 Server: This is the URL for the Icinga 2 virtualization server (e.g. icinga2host.yourdomain.com).
- Icinga 2 API Port: This is the API port, 5665 is the default.
- Icinga 2 API Username and Password: This is your username and password for the API on the Icinga 2 server which is stored in this file /etc/icinga2/conf.d/api-users.conf.
- Trust SSL Certificate on Icinga 2 Server?: Check this box if the Icinga 2 server is configured for secure HTTPS.
- Icinga 2 Server SSL CA Certificate: If SSL, this is the ca.crt location.
- Icinga 2 Server SSL CA Certificate Keystore: If SSL, this is the icinga2-keystore.jks location.
- Icinga 2 Server SSL CA Certificate Keystore Password: If SSL, this is the keystore password.
- Graph Icinga 2 Service Metrics?: Select to include Icinga2 service metrics.
- Sync Interval (in mins): This is the polling interval for collecting monitoring data from the virtual instance and sending it to the GroundWork server. The value is in minutes.
Connection Retries (-1 infinite): This entry is the number of retries for the connection and sets a limit on how many attempts are made after a failure. If you set this to -1 the retrying goes on forever. The number set indicates how many connections are attempted before the connection is left inactive (until you restart it).
Select SAVE. When you choose to save the Cloud Hub connector is assigned an agent ID and that in turn becomes a record locator in Foundation when you begin monitoring.
Then to validate the configuration select TEST CONNECTION which will check if the virtual instance is accessible with the given credentials. If successful you should see Connection successful! at the top of the screen. Select Home to return to the main Cloud Hub panel, and then START (which turns to STOP) for the specific connector to begin the discovery and data collection process.

Figure: Cloud Hub Configuration

2.4 Determining metrics to be monitored

The metrics are set within Icinga 2 and not within the GroundWork Monitoring Cloud Hub interface.

3.0 Unified Monitoring

So how does all this get represented in the unified monitoring context? The data for the monitored services selected are passed to the GroundWork REST API and are directly inserted into the Status and Event Console tables in the GroundWork Foundation database which makes them show up in the UI almost immediately.

3.1 Status view

After starting the connection, in a couple minutes the Status viewer application will display the automatically created host groups corresponding to the views chosen in setup. The monitoring can be adjusted by returning to the Cloud Hub configuration screen and modifying metrics collected (check/un-check) or modifying threshold values.

You may assign the discovered host groups to Custom Groups (e.g. Virtual, Icinga 2) in order to organize the Status display.
You will see the Containers represented as Host Groups, the elements as Hosts, and the metrics are represented as services on the hosts, creating a hierarchy that fits into the GroundWork Monitor UI tree view.
Names of the instances are prefixed with the defined string (e.g. ICINGA2: in example)

In our example, we show linux-servers as the host group, ICINGA2:demo70.groundwork.groundworkopensource.com as the host, and the detail for the service ICINGA2:disk / is displayed as DISK OK - free space: / 19432 MB (68% inode=94%);. In this view you can also see the graphs coming in under Service Availability and Performance Measurement, and the Events being logged at the bottom of the screen.

Figure: Status view

3.2 Event Console

Here in Event Console, we have selected the system Applications filter ICINGA2, which lists events for this application type. From here you can select specific events and apply various actions.

Figure: Event Console, by Application Type (ICINGA2)

3.3 Dashboards

This view displays the Enterprise View dashboard and indicates the host ICINGA2:demo70.groundwork.groundworkopensource.com status as Host Up.

Figure: Icinga 2 Connection - Dashboards, Enterprise View

3.4 NoMa

You can use NoMa to receive alerts and notification. In our example everything has an OK status so no notifications are logged.

4.0 Monitoring Profile for the Icinga 2 Virtual Environment

The management and maintenance for Icinga 2 is within the Icinga 2 console itself. GroundWork treats Icinga 2 as a source of data coming in with no preconceptions, the Foundation database is updated with the console setting.

5.0 Removing Connectors from Monitoring

If you decide you do not want to monitor a particular region, simply navigate to GroundWork Administration > GroundWork Cloud Hub select STOP for the connector, then DELETE. All of the created host groups and the discovered and monitored instances for that region will be deleted from the Foundation database within a few minutes and monitoring access to the region endpoint will cease.

Additionally, see How to remove Cloud Hub hosts in the document How to delete or remove hosts.