Data collector for LDAP

This collector enables administrators to automatically feed iTop with relevant and accurate information collected from a single LDAP Directory.

The Data collector for LDAP is a stand-alone PHP software that connects to a LDAP or AD server, retrieves objects from these environments and synchronises them with iTop's CMDB.

• Connection to LDAP or AD is made through the standard LDAP protocol,
• Retrieval of objects relies on that same protocol,
• Synchronization follows iTop's built-in Data Synchronization mechanism.

Based on LDAP data, the collector focuses on the creation and update in iTop of:

• Persons,
• Users,
• and Users' profiles as an option.

From a technical standpoint:

• The collector can reside on any system with web access to iTop and LDAP access to the LDAP Directory
• The collector is compatible with Windows Active Directory
• The definition of the mapping between LDAP fields and iTop fields is fully configurable.
• The creation of the Synchronization Data Sources in iTop is fully automated.
• Large sets of items can be retrieved through pagination when the LDAP server enforces a limit to the search results.

The collector only synchronizes a limited subset of objects from LDAP (see above).

• Neither the Organizations nor the Locations are synchronized.
• The location of a person is not synchronized
• One collector is collecting data from one single LDAP directory instance only.

Usage of the collector requires you to comply with a few points:

• The LDAP objects synchronized by the collector must exist in iTop (Users or CMDB CIs). This is, by default, the case with iTop's standard data model.
• In order to retrieve LDAP data, you'll need a LDAP access to the Enterprise directory and a read user to access the data.
• From a system standpoint:
  • You'll need to comply with the requirements expressed in the Data collector Base documentation.
  • PHP version should be 8.1, 8.2 or 8.3. These versions include the required support of pagination which is available since PHP 7.3.0.

Simply expand the content of the zip archive into a folder on the machine where the collector will be run.

The configuration of the application is built by concatenating 3 files:

1. <ldap_collector>/collectors/params.distrib.xml that holds entries that are specific to the Data collector for LDAP. It should not be modified.
2. <ldap_collector>/conf/params.distrib.xml that is provided by the collector framework, Data collector Base. It should not be modified.
3. <ldap_collector>/conf/params.local.xml where the collector can be adapted to the specific customer needs.

The conf/params.distrib.xml configuration file lists, amongst others, the parameters that are required for the collector to connect to iTop. Proper ones must be changed through the conf/params.local.xml file.

<?xml version="1.0" encoding="UTF-8"?>
<!-- Default values for parameters. Do NOT alter this file, use params.local.xml instead -->
<parameters>
  <itop_url>https://localhost/iTop</itop_url>
  <itop_login>admin</itop_login>
  <itop_password>admin</itop_password>
  <itop_token/>
  <itop_login_mode/>
 
  ... Other general parameters ...
 
</parameters>

The collectors/params.distrib.xml configuration file holds parameters that must (for some) or can (for others) be changed when configuring the collector, which must be done through the conf/params.local.xml file.

<?xml version="1.0" encoding="UTF-8"?><!-- Default values for parameters. Do NOT alter this file, use params.local.xml instead -->
<parameters>
        <!--  NOTE: ldaphost and ldapport are deprecated in favor of ldapuri -->
        <ldapuri></ldapuri>
        <ldapdn>DC=company,DC=com</ldapdn>
        <ldaplogin>CN=ITOP-LDAP,DC=company,DC=com</ldaplogin>
        <ldappassword>password</ldappassword>
 
        <!-- *** LDAP Class Parameters *** -->
        ...
           List of LDAP classes with their parameters
        ...
 
        <!-- Class collection sequence -->
        ...
           List of classes to collect with their rank in the collection process
        ...
 
 
        <!-- Synchronization parameters -->
        <contact_to_notify></contact_to_notify>
        <synchro_user></synchro_user>
        <!--
             Set a non empty (and unique) prefix if you run several instances of the collector against the same iTop Server
             This is the recommended method to collect data from several LDAP servers. (assign a unique prefix to each "source" LDAP server)
             Note: prefix is defined here since json_placeholders (below) can only reference global parameters
     -->
        <prefix></prefix>
        <json_placeholders>
                <itopperson_synchro_name>$prefix$Synchro LDAP Person</itopperson_synchro_name>
                <persons_data_table>synchro_data_$prefix$ldap_persons</persons_data_table>
                <itopuser_synchro_name>$prefix$Synchro Users from LDAP</itopuser_synchro_name>
                <users_target_class>UserLDAP</users_target_class>
                <users_data_table>synchro_data_$prefix$ldap_users</users_data_table>
                <synchro_status>production</synchro_status>
                <full_load_interval>604800</full_load_interval><!-- 7 days (in seconds): 7*24*60*60 -->
                <!-- Following policies may be: master_locked, master_unlocked or write_if_empty
                     . Use master_locked if you manage the profiles directly in LDAP and let the collector synchronize them in iTop
                     . Use master_unlock if you don't want the list of profiles to be synchronized
                     . write_if_empty if you want the collector to initialize the profiles for each account but let you edit them in iTop after the initial creation of the user
                 -->
                <profile_list_update_policy>write_if_empty</profile_list_update_policy>
        </json_placeholders>
        <!-- Date format for LDAP collector -->
        <date_format>Y-m-d</date_format>
</parameters>        

This set of parameters is required to connect to iTop application or to LDAP environment. Some of them must or may be adjusted to meet customers' own environment.

The parameters that are defined in this section will directly alter the content of the json files used by the collector to build the synchronization data sources in the remote iTop. Some of them must or may be adjusted to meet customers' environments.

Next to the core parameters described here above, the collectors/params.distrib.xml file provides the list of all iTop classes that need to be collected. For each of them, it lists the parameters that should be synchronized within iTop and, for each of them, their counter part in LDAP or AD.

General format is as follows:

<ldapCLASSfilter>ldap_filer</ldapCLASSfilter>
<CLASS_fields>
        <primary_key>ldap_attribute_used_as_a_key</primary_key>
        <itop_attribute#1>ldap_attribute#1</itop_attribute#1>
        <itop_attribute#2>ldap_attribute#2</itop_attribute#2>
        ...
        <itop_attribute#n>ldap_attribute#n</itop_attribute#n>
</CLASS_fields>
<CLASS_defaults>
        <itop_attribute#m>default_value#m</itop_attribute#1>
</CLASS_defaults>

This section defines the list of classes that will be collected and in which order. It enables as well the possibility to deactivate the collection of a class.

<collectors_launch_sequence type="array">
        <!-- Persons -->
        <collector>
                <name>iTopPersonLDAPCollector</name>
                <enable>yes</enable>
                <rank>1</rank>
        </collector>
        <!-- Users -->
        <collector>
                <name>iTopUserLDAPCollector</name>
                <enable>yes</enable>
                <rank>2</rank>
        </collector>
</collectors_launch_sequence>

The launch of the LDAP collector will be driven by the command and parameters defined in the usage section of iTop Data collector base. Once launched, first action of the collector will be to build its collection plan, based on the list of classes that have been enabled in the configuration file. Then,

• Configuration files will be consolidated,
• Synchronisation data sources will be created or updated if required,
• Collection of LDAP classes will be made by connecting to the LDAP environment and extracting data which will be stored under the local “data” directory, in csv format,
• Synchronisation will run and LDAP objects will be pushed to iTop.

Defaults parameters used to collect that class are:

<!-- Parameters for Person synchronization -->
<ldappersonfilter>(objectClass=person)</ldappersonfilter>
<person_fields>
        <!--  Mapping between LDAP fields and iTop Person's object fields -->
        <primary_key>samaccountname</primary_key>
        <name>sn</name>
        <first_name>givenname</first_name>
        <email>mail</email>
        <phone>telephonenumber</phone>
        <mobile_phone>mobile</mobile_phone>
        <function>title</function>
        <employee_number>employeenumber</employee_number>
</person_fields>
<person_defaults>
        <!--  Default values for iTop Person's object fields -->
        <org_id>Demo</org_id>
        <status>active</status>
</person_defaults>

<!-- Parameters for User synchronization -->
<ldapuserfilter>(&amp;(objectClass=person)(mail=*))</ldapuserfilter>
<synchronize_profiles>no</synchronize_profiles>
<itop_group_pattern>/^CN=itop-(.*),OU=.*/</itop_group_pattern>
<user_fields>
        <!--  Mapping between LDAP fields and iTop UserLDAP's object fields -->
        <primary_key>samaccountname</primary_key>
        <login>samaccountname</login>
        <contactid>mail</contactid>
</user_fields>
<user_defaults>
        <!--  Default values for iTop UserLDAP's object fields -->
        <profile>Portal user</profile>
        <!--  Or you define several profiles -->
         <!-- <profile_list>profileid->name:Support agent|profileid->name:Configuration manager</profile_list>  -->
        <language>EN US</language>
</user_defaults>

A user cannot be created in iTop with no profile. It is, therefore, up to the collector to make sure that the attribute profile_list is set when a new user is created. The LDAP collector provides different possibilities to set this attribute, according to the content of the following configuration parameters:

• <synchronize_profiles>
• <user_defaults>/<profile>
• <user_defaults>/<profile_list>

If <synchronize_profiles> is set to ‘yes’:

• The attribute <profile_list> is filled from the LDAP attribute 'memberof', using the regular expression defined by itop_group_pattern to extract the name of the iTop profile or profiles

If <synchronize_profiles> is set to ‘no’ or
If <synchronize_profiles> is set to ‘yes’ and the above computation doesn't extract any profile and leaves the profile_list attribute empty:

• If <user_defaults>/<profile_list> parameter is not set:
  • profile_list is filled with the content of the parameter <user_defaults>/<profile> (*)
• If <user_defaults>/<profile_list> parameter is set:
  • It becomes the default value for profile_list

...
<user_defaults>
   ...
   <profile_list>profileid->name:name_of_profile1|profileid->name:name_of_profile2</profile_list>
</user_defaults>
...

To test and troubleshoot connection problems, use the script ldap-test.php located in the collector/bin folder. The script uses the same parameters as the normal collector, but produces more debug output. To do so, edit the configuration in the file conf/params.local.xml, then launch the test script by typing the following command from the command prompt.

php collectors/bin/ldap-test.php

If you see a message like:

Error - ldap_bind('cn=admin,dc=combodo,dc=com', '*******') FAILED (Can't contact LDAP server).

then something is wrong with the connection to the LDAP server.

1. Check that parameter <ldapuri> is correct. (protocol, host and port)
2. Check that the connection to the server is not blocked by a firewall (You can use the command telnet <host> <port> and see if the connection is established).
3. Check for TSL/SSL problems. If you see the following text in the output of the ldap-test.php script, then the problem is likely related to a TLS certificate:

attempting to connect: 
connect success
TLS: peer cert untrusted or revoked (0x402)
TLS: can't connect: (unknown error code).

The solution is to instruct LDAP to ignore this faulty certificate, by adding the following lines to the LDAP configuration file (see the note below).

# Ignore the server's certificate
TLS_REQCERT never

If the output of the ldap-test.php script contains:

Error - ldap_search('dc=combodo,dc=net', '(objectClass=inetOrgPerson)') FAILED (No such object).

Then check the LDAP query used for retrieving the “contacts”. This query is defined by the two parameters:

    <ldapdn>DC=company,DC=com</ldapdn>
 
    <!-- Parameters for Person synchronization -->
    <ldappersonfilter>(objectClass=person)</ldappersonfilter>

If the LDAP query is correct, you should see an output similar to:

List of the attributes to retrieve (taken from the mapping):
uid,sn,givenname,mail,telephonenumber,mobile,title,employeenumber,memberof
Use --attributes=x,y,z to retrieve x, y and z instead. Use --attributes=* to retrieve all fields.
Debug - ldap_connect('ldaps://customers.combodo.com')...
Debug - ldap_bind('cn=admin,dc=combodo,dc=com', 'xxxxxxx')...
Debug - ldap_bind() Ok.
Debug - ldap_search('dc=combodo,dc=com', '(objectClass=inetOrgPerson)', ['uid', 'sn', 'givenname', 'mail', 'telephonenumber', 'mobile', 'title', 'employeenumber', 'memberof'])...
Debug - ldap_search() Ok.
The LDAP query '(objectClass=inetOrgPerson)' returned 13 elements.
Displaying only 10 elements (use --max-records=xx to change this limit).
------------------------------------------------
LDAP Structure:
Info: when a field is empty on a given record, it is not returned by LDAP.
------------------------------------------------
givenname : bruce
sn        : Lee
uid       : blee
mail      : bruce.lee2@combodo.com
mobile    : 0608080808
------------------------------------------------
givenname : chuck
mail      : chuck.norris@combodo.com
sn        : Norris
uid       : cnorris
------------------------------------------------

The first column of the output is the name of the field in LDAP (all fields returned by the LDAP query are listed) and the second column shows the values of the first record found in LDAP. Based on the values displayed you can complete the configuration of the mapping in the configuration file conf/params.local.xml.

Finally you can test your configuration without importing any data in iTop by running the following command from the command line:

php exec.php --console_log_level=9 --collect_only

This produces an output similar to the one shown below:

Debug - OK, the required PHP version to run this application is 5.3.0. The current PHP version is 7.2.7-0ubuntu0.18.04.2.
Debug - OK, the required extension 'simplexml' is installed (current version: 7.2.7-0ubuntu0.18.04.2 >= 0.1).
Debug - OK, the required extension 'dom' is installed (current version: 20031129 >= 1.0).
Debug - The following configuration files were loaded (in this order):

        1. /opt/dev/ldap-collector/conf/params.distrib.xml
        2. /opt/dev/ldap-collector/collectors/params.distrib.xml
        3. /opt/dev/ldap-collector/conf/params.local.xml

The resulting configuration is:

<?xml version="1.0" encoding="UTF-8"?>
<parameters>
  <itop_url>http://itop-demo/trunk</itop_url>
  <itop_login>admin</itop_login>
  <itop_password>admin</itop_password>
  <console_log_level>6</console_log_level>
  <syslog_log_level>-1</syslog_log_level>
  <max_chunk_size>1000</max_chunk_size>
  <itop_synchro_timeout>600</itop_synchro_timeout>
  <stop_on_synchro_error>no</stop_on_synchro_error>
  <curl_options>
    <CURLOPT_SSLVERSION>CURL_SSLVERSION_SSLv3</CURLOPT_SSLVERSION>
    <CURLOPT_SSL_VERIFYHOST>0</CURLOPT_SSL_VERIFYHOST>
    <CURLOPT_SSL_VERIFYPEER>1</CURLOPT_SSL_VERIFYPEER>
  </curl_options>
  <collect_person_only>no</collect_person_only>
  <ldaphost>192.168.10.13</ldaphost>
  <ldapport>389</ldapport>
  <ldapdn>OU=FGA,DC=combodo,DC=net</ldapdn>
  <ldaplogin>COMBODO\administrateur</ldaplogin>
  <ldappassword>xxxxxx</ldappassword>
  <ldappersonfilter>(objectClass=person)</ldappersonfilter>
  <itop_group_pattern>/^CN=itop-(.*),OU=.*/</itop_group_pattern>
  <person_fields>
    <primary_key>samaccountname</primary_key>
    <name>sn</name>
    <first_name>givenname</first_name>
    <email>mail</email>
    <phone>telephonenumber</phone>
    <mobile_phone>mobile</mobile_phone>
    <function>title</function>
    <employee_number>employeenumber</employee_number>
  </person_fields>
  <person_defaults>
    <org_id>Demo</org_id>
    <status>active</status>
  </person_defaults>
  <ldapuserfilter/>
  <user_id>samaccountname</user_id>
  <user_contactid>mail</user_contactid>
  <synchronize_profiles>no</synchronize_profiles>
  <user_fields>
    <primary_key>samaccountname</primary_key>
    <login>samaccountname</login>
    <contactid>mail</contactid>
  </user_fields>
  <user_defaults>
    <profile>Portal user</profile>
    <language>EN US</language>
  </user_defaults>
  <prefix/>
  <json_placeholders>
    <prefix>$prefix$</prefix>
    <persons_data_table>synchro_data_$prefix$ldap_persons</persons_data_table>
    <users_data_table>synchro_data_$prefix$ldap_users</users_data_table>
  </json_placeholders>
  <ldapfilter>(objectClass=person)</ldapfilter>
</parameters>

Debug - Persons: Mapping of the fields:
   iTop 'primary_key' is filled from LDAP 'samaccountname' 
   iTop 'name' is filled from LDAP 'sn' 
   iTop 'first_name' is filled from LDAP 'givenname' 
   iTop 'email' is filled from LDAP 'mail' 
   iTop 'phone' is filled from LDAP 'telephonenumber' 
   iTop 'mobile_phone' is filled from LDAP 'mobile' 
   iTop 'function' is filled from LDAP 'title' 
   iTop 'employee_number' is filled from LDAP 'employeenumber' 
   iTop 'org_id' is filled with the constant value 'Demo'
   iTop 'status' is filled with the constant value 'active'

Debug - LDAPUsers: Mapping of the fields:
   iTop 'primary_key' is filled from LDAP 'samaccountname' 
   iTop 'login' is filled from LDAP 'samaccountname' 
   iTop 'contactid' is filled from LDAP 'mail' 
   iTop 'language' is filled with the constant value 'EN US'
   iTop 'profile_list' is filled with the constant value 'profileid->name:Portal user'
  
...

You can see the order in which the configuration files were loaded and the resulting configuration.

If the collector does not retrieve all the expected records, this may be due to the server enforcing a limit to the length of the search results. This limit can be overcome by setting the parameter page_size (in the configuration file params.local.xml) to activate the pagination. Set a value less than the maximum number of search results allowed by the LDAP server to ensure that the data will not be truncated.

Question: how can I synchronize iTop with several LDAP servers ?
Answer: You'll need to create several copies of the LDAP data collector: collector-ldap1, …, collector-ldapn and create a specific configuration file for each of them. These configuration files will make sure each collector will connect to a specific LDAP server but to the same iTop instance.

The key, here, is to make sure that the <prefix> parameter is different in each of the configuration files.

In collector-ldapn/conf/params.local.xml put:

<parameters>
        ...
        <ldapuri>ldp://ldap-servern.demo.com:389</ldapuri>
        ...
        <prefix>ldapn_</prefix> <!-- IMPORTANT to have a unique prefix, use only [a-zA-Z0-9_] characters -->
        ...
</parameters>

This will create n independent sets of Synchronization Data Sources:

Question: Can I change the status field's setting in the DataSynchro for Users on iTop ?
Answer: Well, LDAP does not have, out of the box, a simple active/inactive status for a person that could be mapped to the enable/disable status of an iTop User. As a consequence, that attribute is not filled by LDAP: all users are created with the default “enable” status… and left with that same status until the time where the user is removed from LDAP, which triggers its deletion from iTop.