Data collector for CheckMK

with Workaround-Release for Compatibility with CheckMK 2.0

• Collects Server- and PC-Inventory with Brands, Models, OS-Families and -Versions aswell as Softwarepackets
• Uses the iTop SynchroDataSource Feature to synchronize the data with iTop

• 17.4.0: Use this version, if you are running CheckMK 1.x. Ignore the last chapter of this manual. Download
• 22.3.0: Use this version, if you are running CheckMK 2.x. Do not ignore the last chapter of this manual. It is mandatory to create compatibility with CheckMK 2.x. Download

• This collector needs (write-)access to the CheckMK Inventory Files. This implies the collector is running on a machine, which as access to the file system where CheckMK stores its data. You can aswell instruct CheckMK or the underlying server to copy/clone the Inventory Files somewhere else and let the collector do its work there.

• The distinction on whether a file describes a server or a PC depends on your configuration of the collector with regards to filenamepatterns you can specify (more info below).

• PHP Version 5.3.0
• Read access on CheckMK's Inventory Files
• Access to the iTop REST service

• Download one of the two files suiting your needs
• Extract the archive to your filesystem
• Modify the file conf/params.local.xml with respect to your iTop instance (you can use conf/params.distrib.xml as a template)
• Create the file collectors/params.distrib.xml (you can use the file collectors/params.template.xml as a template) and modify the configuration according to your needs

Modify the following parameters in conf/params.local.xml according to your needs:

<itop_url>https://localhost/itop/</itop_url>
<itop_login>admin</itop_login>
<itop_password>admin</itop_password>

The following parameters must be set in conf/params.local.xml aswell.

<default_org_id>Demo</default_org_id>
<check_mk_dir>/var/lib/check_mk/inventory</check_mk_dir>
<type_mapping type="array">
  <pattern>/.*pc.*/PC</pattern>
  <pattern>/.*srv.*/Server</pattern>
</type_mapping>
<import_software_packages>yes</import_software_packages>

The following optional boolean parameters might be set according to your needs in conf/params.local.xml aswell:

<use_network_hostname>false</use_network_hostname>
<skip_gz>true</skip_gz>
<skip_dot>true</skip_dot>

It might be of interest for your, to specify default values for certain fields, which otherwise won't be filled due to the data delivered by CheckMK. You can do so by editing collectors/params.distrib.xml as follows. (And you can overwrite those defaults in conf/params.local.xml.)

<default_fields type="hash">
  <field_name>field value</field_name>
  <field_name2>another field value</field_name2>
</default_fields>
<default_fields_sw type="hash">
  <status>active</status>
</default_fields_sw>

Default fields for servers an PCs can be specified in <default_fields>. Default fields for software packages can be specified in <default_fields_sw>

The data delivered by CheckMK do not always fit the iTop Data Structures (which you might have modified). A mapping table is used to convert data acccording to your datamodel. This collector uses this feature for Brands, Models, OS-Families and OS-Versions.

Default Mapping Tables are located in collectors/params.distrib.xml. You can overwrite those via conf/params.local.xml. The file collectors/params.template.xml contains more examples and more information.

Example:

<osfamily_mapping type="array">
  <pattern>/.*linux.*/Linux</pattern>
  <pattern>/.*windows.*/Windows</pattern>
  <pattern>/.*mac os.*/Mac OS</pattern>
  <pattern>/.*solaris.*/Solaris</pattern>
  <pattern>/.*bsd.*/BSD</pattern>
  <pattern>/.*/$1$s</pattern>
</osfamily_mapping>

This example considers Linux, Windows, Mac, Solars and BDS OS-Families normalizing their naming. With <pattern>/.*/$1$s</pattern> you are instructing the collector to keep all non-fitting names to be preserved as-is. Always keep this rule, but always keep this rule the last one.

The first time you are running the collector, it is recommended to run the collector as follows:

php exec.php --configure_only

By running the collector with the parameter “–configure_only” you are instructing it to only connect to iTop and create a new SynchroDataSource as specified by the collector.

To only collect data (but not synchronize with iTop), use the following command:

php exec.php --collect_only

aus.

This instructs the collector to collect the data delivered by CheckMK, transform and output the transformed data into the folder data/ into csv-files. This might be useful to manually check the transformed data for missing or wrongly transformed data.

Finally, if all the data is transformed correctly, you now can run

php exec.php --synchro_only

to synchronize the data with iTop for the first time, but you should run –collect directly before that command again.

If everything works, you can use the command

php exec.php

to do all of the above with one command.

More information on how and which runtime parameters for the collector can be used you can find at itop-data-collector-base

Server and PCs contain the same types of data.

If activated Software will be collected and synchronized as objects in the iTop Software Catalogue.

All Softwareinstanzen like PCSoftware, WebServer, DBServer, Middleware oder OtherSoftware contain the same set of fields to be synchronized.

This release is a workaroundrelease to temporarely create compatibilty with how CheckMK 2.x now stores data in Inventory Files.

In this chapter you will learn how to make the collector compatible with CheckMK 2.x

CheckMK 1.x used Python 2. For that reason, the Inventory Files contain serialized objects in Python2 Object Notation (P2ON). The collector compatible with CheckMK 1.x contains a custom parser to handle P2ON.

With the release of CheckMK 2.x they moved to Python 3. Thus, the Inventory Files now contain P3ON. The custom parser for P2ON is not able to parse those files on number of occassions (e. g. special Unicode chars).

This is the reason, ITOMIG has (temporarely) implemented a workaround to still be able to use the collector with CheckMK 2.x.

The workaround is based on letting Python 3 do the work to change the format of the Inventory Files from P3ON to valid, simple JSON, which PHP/the collector is able to hanle easily.

It is assumed, that

• you have read and understood the manual above (how the collector basically works) and that you have configured it according to your needs,
• you are using CheckMK 2.x,
• there is Python 3 installed on the machine, where the collector will be residing,

For the following example we furthermore assume, that

• the Inventory Files of CheckMK are saved in /var/lib/checkmk/inventory_files/,
• the collector resides in the folder /etc/itop/collectors/collector.checkmk/ and
• the Cronjob for running the collector is configured to run once every day at 10:23 a.m.

• Inside the file <collectorpath>/workaround/pon2json.py modify the values of the two top variables according to your needs, e. g.:

path_original = "/var/lib/checkmk/inventory_files/"
path_output = "/etc/itop/collectors/collector.checkmk/json_files/"

• Pay attention, the folder “json_files” really exists and is writable for the cronuser.
• Edit the file <collectorpath>/collectors/params.distrib.xml like so: <check_mk_dir>/etc/itop/collectors/collector.checkmk/json_files/</check_mk_dir>, such that the collector is configured to read the JSON-Files, which are already transformed.

• Move inside the folder workaround folder inisde the collector and on the commandline and run the following command python3 pon2json.py. The expected output of the script looks like this:

Inputfolder: /var/lib/checkmk/inventory_files/
Files found:
['server-01', 'pc-01', 'example.txt']
Files to ignore:
['example.txt']
Files to convert:
{'server-01', 'pc-01'}
server-01 converted...
pc-01 converted...
Outputfolder: /etc/itop/collectors/collector.checkmk/json_files/
Done

• Now check, there really are transformed JSON-Files written to the folder “json_files” and that the content looks reasonable.
• Afterwards move inside the root-folder of the collector and run php exec.php --collect_only.
• Check for new/updated csv-files in the folder /data/ of your collector and that the content looks reasonable.
• Finally, run a complete synchronisation by running the command php exec.php from the root folder of your collector. In iTop, check, if Models, Brands, Server, PCs and so on are created according to your configuration and the Inventory Files.

• Create a new Cronjob additional to the Cronjob already existing for running the collector.
  • The content should look something like this: 17 10 * * * /usr/bin/python3 /etc/itop/collectors/collector.checkmk/workaround/pon2json.py >> /var/www/html/itop/log/checkmk-konverter.log 2>&1
  • With this Cronjob you are making sure, that the actual collector always works on freshly transformed Inventory Files, because the collector runs 5 minutes after the transformation from P3ON to JSON.

• Pay attention for “Traceback”-Messages inside the converter log. If they appear an error happened during the transformation from P3ON to JSON.
• The converter assumes, all applicable Inventory Filenames in the Inventory Folder do not contain any dots. E. g., they should be named “pc-01” and son on. Files with dots in the filename are ignored. Other folders inside the Inventory Folder aswell.