Automated Agent Registration

h6. Contents

This page references the Automated Agent Registration for GDMA.

{toc:minLevel=4|maxLevel=5|printable=false}

h5. General Description

The GDMA installer asks for an Automated Agent Registration username/password at install time. (The defaults presented to the user are blank, but {{gdma}}/{{gdma}} will work in a fresh GWMEE 6.7.0 install out of the box. Changing the password on the server, in Administration>User Management, is recommended for customers before deploying GDMA.) If both values are given, then the auto-registration is enabled on that GDMA client. If one or both are blank, then the auto-registration is disabled and the old auto-configuration mechanism will still be in play, albeit perhaps at a slower pace than before (see the hour delay noted below).

This new {{gdma}} user on the server has limited (generally read-only) privileges for accessing most server functions, and those restrictions should be left in place. It is intended that this username be used only for this Automated Agent Registration purpose, partly so that if any trouble arises with it, the account can be disabled without impacting any other aspect of the monitoring.

The GDMA client installer also asks for host and service profile names. Both are optional. The installer suggests a standard default value for the host profile, which happens to be the same as the initial default we have configured on the server side. That default is {{gdma-linux-host.xml}} or equivalent for other platforms ({{gdma-windows-host.xml}}, {{gdma-solaris-host.xml}}, {{gdma-aix-host.xml}}). If no service profile is specified, then no extra service profile will be applied beyond any service profiles referenced directly by the host profile.

The new values requested by the GDMA client installer are stored on the client in {{gdma/config/gdma_auto.conf}}:

{code}
Auto_Register_Host_Profile = ""
Auto_Register_Pass = ""
Auto_Register_Service_Profile = ""
Auto_Register_User = ""
{code}
These settings can be adjusted after installation, if need be. Like anything else in the {{gdma_auto.conf}} file, modifying values there will in due course cause the client to recognize a configuration change and attempt to reload its own configuration --- now in up to three files: {{gdma_auto.conf}}, {{gdma_override.conf}}, and the {{gwmon_\{hostname\}.conf}} file.

The {{gdma/config/gdma_override.conf}} file is new with the GDMA 2.3.0 release. It is dynamically created as the place where the poller will store the hostname returned by the server, after a successful auto-registration request. The poller uses a {{Forced_Hostname}} option to save this centrally-agreed-upon hostname. This gets picked up by the spooler as well, so both poller and spooler know the registered name of the host. Any use of the {{Forced_Hostname}} option in the {{gdma_auto.conf}} file will be overridden by the value in the {{gdma_override.conf}} file, if the latter file is present.

When the client makes an auto-registration call to the server, it passes along a small amount of information about the client system, including the configured {{Auto_Register_Host_Profile}} and {{Auto_Register_Service_Profile}} values. The {{foundation/scripts/registerAgentByProfile.pl}} script on the server is called to validate the host attributes, add the host to Monarch, build externals for the host (based on whatever externals were attached to the host profile, or to the services attached to the service profile(s)), and return the final hostname. Building externals means that the associated host and service externals are collected together, processed via possible macro substitution, and written to a location where the client can pick them up.

In practice, a green system operates in the following way. An unconfigured poller will reach for a local copy of the host-config file, find none, and go to sleep for 10 minutes (configured in {{gdma_auto.conf}} as the {{Poller_Proc_Interval}}). On waking up, when it again sees it still cannot find its config file, if it has auto-registration credentials, it will do the auto-registration thing; otherwise, it will fall back to the old auto-configure protocol. All the auto-registration thing does is to make a call to the server, and look at the results. If the results include a hostname, the poller stores the hostname in the {{gdma_override.conf}} file. At that moment, it still doesn't have a host config file in hand. On the next cycle, it should pull the externals file from the GroundWork server, save it locally, and start monitoring the client machine.

The spooler, in its own cycle time, eventually sees that the configuration files have changed, and re-reads them. Then it will see the forced hostname and adapt to it.

When an auto-registration request comes into the server, it first tries to look up in Monarch to see if the host is already registered. There is a special Perl package installed ({{foundation/scripts/AutoRegistration.pm}}) that implements a pass of possible host-attribute recoding before this lookup. If that lookup fails, a second possible recoding (presumably with different logic this time) and lookup is attempted. If both passes fail and the package has not concluded that the attributes are totally invalid, the host will be added to Monarch.

The second pass of recoding allows IP => hostname lookups based on specific addresses listed in the server {{config/register_agent.properties}} file, for cases where DNS just isn't operating right on the client and there's no way to fix that easily. (We've tried to cover all the bases, for situations seen in the field. If we've missed any, let us know and we'll see about folding more capability into the next version.)

{note}Auto-registered hosts will begin monitoring as soon as they receive their own host configuration file from the server. They will then start sending on the results of those checks to the GroundWork server. At this point, however, Nagios won't know of the existence of such hosts, and it will casually drop those results on the floor (probably with a notice to that effect in the {{nagios/var/nagios.log}} file). Results from such hosts will continue to be ignored until the administrator (or perhaps some automated process) runs a Commit operation.{note}

h6. Additional Information

The {{config/register_agent.properties}} file on the server contains a number of configuration options to control the server-side scripting.

The {{AutoRegistration.pm}} module on the server contains documentation regarding the internal routines, which can help both with understanding its actions and for making possible modifications:
{code}
/usr/local/groundwork/perl/bin/perldoc /usr/local/groundwork/foundation/scripts/AutoRegistration.pm
{code}

h5. Initial Setup

# Change the password of the {{gdma}} user from the default value that is shipped in the standard release (Administration > User Management > User Management > Search Users > gdma > Edit user > Change password). This is a basic security measure you should take to protect your site.
\\
\\ !aar_1.jpg|border=1!
# Enable externals within Monarch, so you have access to all the externals-related screens and options (Configuration > Control > Setup > Enable externals > checkbox > Save). A variety of screens will now display extra tabs and navigational links allowing you to edit and manage host and service externals.
\\
\\ !aar_2.jpg|border=1!
# Add an "Auto-Registration" hostgroup (without the quotes), under Configuration > Hosts > Host groups > New. You can choose a different name, if you want, but then you will also need to modify the {{default_hostgroup}} setting in the {{/usr/local/groundwork/config/register_agent.properties}} file.
\\
\\ !aar_3.jpg|border=1!
# Add an "auto-registration" Monarch configuration group (without the quotes), under Configuration > Groups > New. You can choose a different name, if you want, but then you will also need to modify the {{default_monarch_group}} setting in the {{/usr/local/groundwork/config/register_agent.properties}} file.
\\
\\ !aar_4.jpg|border=1!
# In the Detail screen for the new Monarch configuration group, set the Build folder to {{/usr/local/groundwork/apache2/htdocs/gdma}} so externals files will be placed where GDMA clients will be looking to pick them up. Save this change.
\\
\\ !aar_5.jpg|border=1!
# It is generally recommended that you add the "Auto-Registration" hostgroup to the "auto-registration" Monarch configuration group (Configuration > Groups > Groups > auto-registration > Detail > Hosts). This will simplify the actions when new hosts are added, by allowing the individual hosts to be indirectly associated with the Monarch configuration group via the hostgroup, instead of being assigned directly to the Monarch configuration group. Make adjustments as needed if you use alternative hostgroup or Monarch configuration group names.
\\
\\ !aar_6.jpg|border=1!
# Add new host profiles and related objects, under Configuration > Profiles > Profile importer > Import > GDMA.  Select the four new host-profile files listed here, and click Import at the bottom of the screen. This will bring in corresponding new host profiles and related host templates, service profiles, service templates, services, host and service externals, commands, performance configuration entries, and a few other minor objects.
\\
\\
{{gdma-aix-host.xml}}
{{gdma-linux-host.xml}}
{{gdma-solaris-host.xml}}
{{gdma-windows-host.xml}}
\\
\\ !aar_7.jpg|border=1!
# Edit each of the new host templates that just got imported, to add one or more default contact groups. Navigate to Configuration > Hosts > Host templates > Modify, then select each one of the host templates listed here and establish the basic contact group(s) you will want for hosts using this host template (perhaps just {{nagiosadmin}} at this point, as a kind of safe default). Save each host template after making this change. The setting here can be overridden later on at other levels, but this provides at least a standard setting for all auto-registered hosts.
\\
\\
{{gdma-aix-host}}
{{gdma-linux-host}}
{{gdma-solaris-host}}
{{gdma-windows-host}}
\\
\\ !aar_8.jpg|border=1!
# The new host profiles you just loaded, listed below, will form the basis for configuring new auto-registered GDMA hosts running the respective operating-system types. You are not required to use these profiles; the GDMA client installer allows you to select an alternate host profile, and an optional additional service profile, to be applied on the server to the auto-registered client's configuration. See also the {{default_host_profile}} setting in the {{/usr/local/groundwork/config/register_agent.properties}} file.
\\
\\
{{gdma-aix-host}}
{{gdma-linux-host}}
{{gdma-solaris-host}}
{{gdma-windows-host}}
\\

h5. FAQs

* *Q:* Which GroundWork Monitor server is the auto-registration request sent to?
* *A:* It's the same server as mentioned as the first value of the {{Target_Server}} option in the {{gdma/config/gdma_auto.conf}} file on the client. See the [GDMA>Configuration Reference|Configuration Reference] page in the Bookshelf. 
\\
\\
* *Q:* What happens during auto-registration if the GDMA client machine is already present in Monarch, and doesn't need to be freshly added to the configuration?
* *A:* The server makes up to two separate attempts to look up the host within Monarch, to find it using possibly different host attributes. If one of those attempts succeeds, the existing hostname in Monarch will be returned to the client as its official, final hostname. But even in this case, the server attempts to ensure that the desired host profile, service profile, hostgroup, and Monarch configuration group assignments are made to this host.
\\
\\
* *Q:* My GDMA client cannot reliably determine its own hostname to send to the server, because DNS is not working at the client. What can I do?
* *A:* Many failures of this type can be corrected by the server during the auto-registration process. Logic in the {{AutoRegistration.pm}} module looks at the IP address sent by the client as well, and tries to determine a correct match between the IP address and the hostname before settling on a final hostname to add to Monarch and to publish back to the client machine. See also the {{<hardcoded_hostnames>}} option in the {{config/register_agent.properties}} file, for handling particularly pernicious problems.
\\
\\
* *Q:* I don't like the form of the hostname that my client machines get registered under. How can I change it?
* *A:* An out-of-the-box setup will cause auto-registered hosts to have fully qualified hostnames within GroundWork Monitor. This form was chosen to provide the least chance of ambiguity in large installations, especially those that serve multiple domains that might otherwise have hostname collisions. But it is not a requirement of operation.
** The client tries to determine its own hostname using standard heuristics, and it sends that hostname to the server as part of the auto-registration request. The server may or may not accept that hostname as-is, but in any case, the server's decision is final. There are two controls, and possibly other settings, to play with.
** On the client side, you can set the {{Forced_Hostname}} option in the {{gdma/config/gdma_auto.conf}} file. See the documentation for this option in the [GDMA>Configuration Reference|Configuration Reference] page in the Bookshelf.
{note}With auto-registration in play, this option in the {{gdma_auto.conf}} file really only affects the name passed to the server by the client during its auto-registration attempt. Afterward, the value of this option dynamically saved by the client in the {{gdma_override.conf}} file will rule.{note}
{note}The {{Use_Long_Hostname}} option will have no practical effect when auto-registration is in effect. The client always attempts to send a fully-qualified hostname to the server as part of an auto-registration attempt, to give the server the best chance of accurately identifying the client.{note}
** On the server side, you can adjust the {{hostname_qualification}} option in the {{config/register_agent.properties}} file, to globally select long or short names, or to pass the decision to your own custom logic. See the documentation for that option in the config file.
** If the hostname is being determined by the {{<hardcoded_hostnames>}} section of the {{config/register_agent.properties}} file, that will be the place you will need to change it.
** In the extreme, if you need some kind of specialized hostname determination that is not covered by the options above, you can modify the logic by which the server chooses the client hostname. You might do this, for instance, if you want to use unqualified hostnames for most of your machines, and fully qualified hostnames only for machines on particular subnets. The client sends multiple attributes to the server as part of an auto-registration request: the type of agent making the request (e.g., "GDMA"), its own hostname (as best it knows it), its IP address, its MAC address, and the generic type of operating system running on the host. Custom logic can use any or all of these attributes to make a final determination of the client hostname.
** To implement custom server logic in this area, you will need to:{html}<ul>{html}{html}<li>{html}Copy the {{foundation/scripts/AutoRegistration.pm}} file into some sibling file of your own name (e.g., {{AcmeCorporationAutoReg.pm}}).
** Change the {{package}} declaration in your file to reflect that name:
{code}
package AcmeCorporationAutoReg;
{code}
** Change the logic in the {{soft_recode_host_attributes()}} and/or {{hard_recode_host_attributes()}} routines as you need.
** Edit the documentation in the file at the end of your own version of this package to reflect your logic changes.
** Change the {{customer_network_package}} option in the {{config/register_agent.properties}} file to name your package instead of the {{AutoRegistration}} package.
{code}customer_network_package = "AcmeCorporationAutoReg"{code}
** This last step will put your package into play.
{note}GroundWork Monitor will overwrite the {{AutoRegistration.pm}} file in future releases, which is part of why you are advised to change the filename and package name if you change the content.{note}
{note}You should be aware that both client-side GDMA and the server-side scripting always lowercase whatever hostname they use, to provide consistency in the face of potential confusion. This behavior is not configurable, as it just causes too much headache to loosen up on this convention.{note}
* *Q:* I don't like the form of the hostname that one of my client machines has already been registered under. How can I change it?
* *A:* You can rename the host within Monarch (*Configuration*>*Hosts*>\{hostgroup\}>\{host\}>*Detail*>*Rename*). After doing so, go to *Configuration*>*Control*>*Build externals*, to rebuild the externals file using the modified hostname. After some time, the GDMA client will recognize that its externals file is no longer available on the server under the old name, and that will eventually cause it to exercise the auto-registration logic. At that time, the client should be recognized by the auto-registration request as being already in Monarch (given that the IP address has not changed, even though the hostname did), and the new name will be returned to the client for further operations there.
** If need be, you can speed up this process by bouncing GDMA on the client after the externals are rebuilt on the server. (See the [GDMA>GDMA Quick Start|GDMA Quick Start] page in the Bookshelf for the commands to do this on each platform.) It will take about 10 minutes (one {{Poller_Proc_Interval}}) for the auto-registration to occur on the client.
** Of course, you will need to run a Commit operation (*Configuration*>*Control*>*Commit*) for Nagios to understand that the hostname has changed, so it doesn't start dropping the data for this host on the floor.
** Renaming an existing host like this is subject to a caveat, namely that in the current release of GroundWork Monitor, performance data collected under the old hostname will not be transferred to the new hostname. The old data still exists, but it is stored in RRD files using the old hostname, while new performance data will be stored in RRD files using the new hostname. We don't currently have a utility to merge the old data into the new RRD files.
\\
\\
* *Q:* What configuration objects are associated with an auto-registered host?
* *A:* Multiple connections will be made during auto-registration:
** Common practice is to have the host profile specified by the client be assigned and applied to the host. If the client does not specify a host profile, the server will assign a host profile based on the {{default_host_profile}} option in the {{config/register_agent.properties}} file. The server can modify these standard choices via the {{host_profile_to_assign()}} routine in a customized version of the {{AutoRegistration.pm}} module.
** Any service profiles associated with the applied host profile will be applied to the host.
** The client can optionally specify one additional service profile to be applied to the host.
** Host and service externals from the host profile and from the services applied to the host will be copied to the configuration for the host. This is a critical link in how the individual GDMA options and service checks for the host are made known to the client to run its monitoring commands.
** The host will become a member of the hostgroup listed as {{default_hostgroup}} in the {{config/register_agent.properties}} file. If a more-complex setup is required, a customized version of the {{AutoRegistration.pm}} module can be used to return a list of hostgroups in the result of the {{hostgroups_to_assign()}} routine. If you truly don't want any auto-registered host to be a member of any hostgroup (probably not a practical setup), that can also be accomplished by having your custom version of the {{AutoRegistration.pm}} module have no {{hostgroups_to_assign()}} routine.
** If the host is not already a member of some Monarch configuration group, the host will become a member of the Monarch configuration group listed as {{default_monarch_group}} in the {{config/register_agent.properties}} file. This standard behavior can be modified via a customized version of the {{AutoRegistration.pm}} module, where the {{monarch_groups_to_assign()}} routine can be used to determine a list of Monarch groups to which the host should be assigned.
** Also see the {{assign_monarch_groups_to_existing_group_hosts}} option in the {{config/register_agent.properties}} file, if you need to allow a host which is already assigned to some Monarch configuration group to also be assigned to the Monarch configuration group(s) determined by the auto-registration logic.
{note}The host must be assigned to at least one Monarch group in order to have externals built and published on the server, where the client can subsequently find them. Some customers might want to create and distribute the client host-configuration files via some other mechanism. A future version of GDMA will include the ability to assign an empty string to the {{default_monarch_group}} option in the {{config/register_agent.properties}} file, to suppress the assignment of the host to a Monarch group, for all hosts. In the meantime, the same effect can be achieved in a custom version of the {{AutoRegistration.pm}} module by either having the {{monarch_groups_to_assign()}} routine return an empty list of Monarch groups, or having no {{monarch_groups_to_assign()}} routine in your package.{note}
** Administration of the Monarch configuration group can be made easier by manually assigning at least one hostgroup the host will be a member of to the Monarch group. Then all hosts that are auto-registered and that are a member of such a hostgroup won't have to be directly assigned to the Monarch group; the indirect association will suffice. This is generally the preferred setup, as it keeps the configuration much simpler.
\\
\\
* *Q:* GroundWork's standard profiles provide a basic set of monitoring checks for each platform (type of operating system). How do I customize the set of checks used by GDMA clients?
* *A:* Use custom profiles instead of the standard profiles that GroundWork provides. You can adjust the sets of checks however you like, selecting the sets of host and service checks you are most interested in.
\\
\\
* *Q:* My monitoring-check definitions need to be written in a generic form so they apply across many hosts. But then how do I get them automatically customized for each particular host?
* *A:* Host and service externals can reference several different types of macros, and have those references be expanded when the externals files are built. Macros are currently expanded in the following order:
** Nagios user-resource macros (the macros named as {{USER1}} through {{USER32}} in the *Configuration*>*Control*>*Nagios resource macros* screen).
** Monarch configuration group macros (created in the *Configuration*>*Groups*>*Macros* screen, and associated with groups and possibly redefined in the *Configuration*>*Groups*>*Groups*>\{groupname\}>*Detail*>*Macros* screen). Since a host can be a member of multiple configuration groups arranged in parent-group/sub-group hierarchies, a standard order has to be defined in which such macros are applied. The rule is that Monarch configuration group macros are substituted in ascending order from subgroups to parent groups associated with the given host. This allows the most-specific value of a given macro (generally associated with a subgroup) to be used in preference to a more-general value of the same macro name (generally associated with a parent group, and used as a default value).
** Certain standard macros: {{HOSTNAME,}} {{HOSTADDRESS}}, and {{HOSTALIAS}}.
\\
\\
In all of these cases, the macro references are recognized by the presence of both leading and trailing "{{$}}" characters around the macro names. So they would appear in your raw externals as references such as:
\\
\\
{code}
$USER32$
$MYGROUPMACRO$

\\
\\
* *Q:* Are there other improvements in this area provided by the GWMEE 6.7.0 release?
* *A:* Building externals is much more sophisticated now. By default, it compares the old and new payload of each generated externals file, and only updates the timestamp of the file if the content has changed. This should cut down dramatically on the amount of noise in the event system from GDMA clients notifying the server of supposed configuration changes. Also, we have improved the code in some places to take very targeted action and only rebuild externals for a few hosts that might actually need it, so we no longer build indiscriminately for all hosts.
\\
\\
* *Q:* How can I disable Automated Agent Registration on a given GDMA client machine?
* *A:* Disabling auto-registration for an individual host can be effected on the client side by editing the {{gdma/config/gdma_auto.conf}} file and either setting the auto-registration access credentials to empty strings, or just commenting out those lines in the file. Such a setup will instead invoke the older auto-configure protocol if the client cannot find its own host-configuration file on the server.
\\
\\
{code}
Auto_Register_Pass = ""
Auto_Register_User = ""
{code}
* *Q:* How can I completely disable Automated Agent Registration?
* *A:* There are two simple server-side controls that will stop all Automated Agent Registration requests from succeeding. Either one alone suffices to disable this facility.
** You can change the password for the auto-registration user that the clients were installed with, without telling the clients the new password.
** In the {{config/register_agent.properties}} file, you can set:
\\
\\
{code}enable_processing = no{code}
* *Q:* Is auto-registration of hosts with IPv6 addresses supported?
* *A:* Not yet, in this release. Some initial support is present, but not to the point where GroundWork considers it to be functional and tested. This capability will evolve in future releases.

h5. Troubleshooting

* If auto-registration is completing but monitoring data from GDMA clients is obviously not being processed by the server, perhaps the problem is that the server does not have the Bronx listener port (by default, port 5667) open to the clients. See the {{listener_port}} value in the {{config/bronx.cfg}} file on the server, and the {{Spooler_NSCA_Port}} value in the {{gdma/config/gdma_auto.conf}} file on the GDMA client. These values must match, and you must not have a firewall (or packet filtering such as iptables on the server) blocking this port.
{html}<p></p>{html}The mechanism for allowing access to a given server port will depend on the particular Linux distribution you are running on the server. On RHEL 6.3, for instance, you would need a line such as the following in the {{/etc/sysconfig/iptables}} file, and then have iptables restarted to pick up this configuration:
{code}
-A INPUT -m state --state NEW -m tcp -p tcp --dport 5667 -j ACCEPT
{code}
* If you think that auto-registration is not doing what it ought to, first look in the server-side log file:
{code}
/usr/local/groundwork/foundation/container/logs/register_agent.log
{code}{html}<!-- illegal HTML here, but Confluence converts it to valid HTML; it's the only way to get reasonable vertical spacing here --></p>{html}This pathname is configured in the {{config/register_agent.properties}} file on the server, and listed also in the {{/etc/logrotate.d/groundwork}} file to enable regular log rotation.
* Auto-registration error messages listed in the server log are generally also returned to the GDMA client, and will be recorded in the client's poller log file if logging is enabled on the client by enabling it in the {{gdma/config/gdma_auto.conf}} file:
{code}
Enable_Local_Logging = "on"
{code}
If the client is having trouble contacting the server in the first place, that may not be evident without enabling logging on the client and looking in the poller log file.
{html}<p></p>{html}If you do enable logging on the client, it is highly recommended that you disable it once you have diagnosed and fixed the problem, because the current version of GDMA never rotates its log files.
* If an auto-registration request doesn't yield any positive results, the poller won't keep banging on the server every iteration. It waits at least an hour before trying again. This interval is currently hardcoded. Due to slight delays in processing a single auto-registration request, in practice this means it won't try again for 70 minutes.
* If you want to speed things up for testing, especially to avoid the initial 10-minute wait before the first auto-registration attempt after the poller starts and finds it has no local copy of the host configuration file, you will probably want to set {{Poller_Proc_Interval}} to {{60}}. If you change this value for testing, be sure to change it back (typically to {{600}}) for production use\! If auto-registration fails on a given attempt and you don't want to wait the extra hour for the following attempt, just bounce GDMA on the client machine. The [GDMA>GDMA Quick Start|GDMA Quick Start] page describes the various platform-specific commands used to start and stop the GDMA client.