Data collector for LDAP

This stand-alone application collects information from a single LDAP Directory in order to automatically synchronize the persons and the users in iTop.

Main functions:

• Automatic creation and update of Persons and Users in iTop based on LDAP data.
• Automatic assignment of Profiles to Users based on LDAP groups (this is optional).

Technical aspects:

• The collector can reside on any system with web access to iTop and LDAP access to the LDAP Directory
• The collector is compatible with Windows Active Directory
• The definition of the mapping between LDAP fields and iTop fields is fully configurable.
• The creation of the Synchronization Data Sources in iTop is fully automated.
• Starting with version 1.2.6, support of pagination for retrieving large sets of items when the LDAP server enforces a limit to the search results.

• The current version is synchronizing neither the Organizations nor the Locations.
• The location of person and the manager of a person are not synchronized.
• One collector is collecting data from one single LDAP directory instance only.
• Date format of the source data must be YYYY-MM-DD (hh:mm:ss)
• Support of pagination requires PHP 7.3.0 or newer.

• PHP (command line interface), version 5.3.0 up to 7.4 with the php-ldap, php-dom and php-simplexml extensions installed. PHP 7.3 or newer is required for the support of pagination.
• An LDAP access to the Enterprise directory and a read user to access the data.
• An HTTP/HTTPS access to the iTop web services (REST + synchro_import.php and synchro_exec.php)
• + Data collector Base requirements.

• Expand the content of the zip archive “ldap-data-collector” in a folder on the machine that will run the collector application. This machine must have an LDAP access to the Enterprise directory and a web access to the iTop server.
• create the file conf/params.local.xml to suit your installation, supplying the appropriate credentials to connect to LDAP server and iTop.

By default this file should contains the values used to connect to the LDAP server and to the iTop server:

params.local.xml

<?xml version="1.0" encoding="UTF-8"?>
<!-- conf/params.local.xml - your specific configuration parameters -->
<parameters>
  <itop_url>http://localhost/</itop_url>
  <itop_login>admin</itop_login>
  <itop_password>admin</itop_password>
  <contact_to_notify>john.doe@demo.com</contact_to_notify>
  <synchro_user>admin</synchro_user>
  <ldapuri>ldap://localhost:389</ldapuri>
  <ldapdn>DC=company,DC=com</ldapdn>
  <ldaplogin>CN=ITOP-LDAP,DC=company,DC=com</ldaplogin>
  <ldappassword>password</ldappassword>
  <page_size>1000</page_size>
  <!--
    Set a non empty (and unique) prefix if you run several instances of the collector against the same iTop Server
    This is the recommended method to collect data from several LDAP servers. (assign a unique prefix to each "source" LDAP server)
    Note: this prefix can be set but do not touch the one inside json_placeholders
    -->
  <prefix></prefix>
  <json_placeholders>
    <full_load_interval>604800</full_load_interval><!-- 7 days (in seconds): 7*24*60*60 -->
    <users_target_class>UserLDAP</users_target_class>
    <synchro_status>production</synchro_status>
  </json_placeholders>
</parameters>

The default values for the configuration of the data collection is defined in the file collectors/params.distrib.xml. This configuration defines which LDAP queries are executed on the LDAP server to retrieve the data, how to map the LDAP fields with the iTop fields and some default values for the iTop fields.

The configuration looks as follows:

<parameters>
        <!-- Parameters for Person synchronization -->
        <ldappersonfilter>(objectClass=person)</ldappersonfilter>
        <person_fields>
                <!--  Mapping between LDAP fields and iTop Person's object fields -->
                <primary_key>samaccountname</primary_key>
                <name>sn</name>
                <first_name>givenname</first_name>
                <email>mail</email>
                <phone>telephonenumber</phone>
                <mobile_phone>mobile</mobile_phone>
                <function>title</function>
                <employee_number>employeenumber</employee_number>
        </person_fields>
        <person_defaults>
                <!--  Default values for iTop Person's object fields -->
                <org_id>Demo</org_id>
                <status>active</status>
        </person_defaults>
        <!-- Parameters for User synchronization -->
        <collect_person_only>no</collect_person_only>
        <ldapuserfilter>(&amp;(objectClass=person)(mail=*))</ldapuserfilter>
        <synchronize_profiles>no</synchronize_profiles>
        <itop_group_pattern>/^CN=itop-(.*),OU=.*/</itop_group_pattern>
        <user_fields>
                <!--  Mapping between LDAP fields and iTop UserLDAP's object fields -->
                <primary_key>samaccountname</primary_key>
                <login>samaccountname</login>
                <contactid>mail</contactid>
        </user_fields>
        <user_defaults>
                <!--  Default values for iTop UserLDAP's object fields -->
                <profile>Portal user</profile>
                <language>EN US</language>
        </user_defaults>
</parameters>

<person_fields>
  <primary_key>samaccountname</primary_key>
  <name>sn</name>
  <first_name>givenname</first_name>
  <email>mail</email>
  <phone>telephonenumber</phone>
  <mobile_phone>mobile</mobile_phone>
  <function>title</function>
  <employee_number>employeenumber</employee_number>
</person_fields>

<person_defaults>
  <org_id>Demo</org_id>
  <status>active</status>
</person_defaults>

<user_fields>
  <primary_key>samaccountname</primary_key>
  <login>samaccountname</login>
  <contactid>mail</contactid>
</user_fields>

<user_defaults>
  <profile>Portal user</profile>
  <language>EN US</language>
</user_defaults>

Those parameters can be redefined in the file conf/params.local.xml in order to take into account your specific needs. (For instance the mapping between iTop and LDAP attributes)

<user_defaults>
  <profile_list>profileid->name:name_of_profile1|profileid->name:name_of_profile2</profile_list>
  ...

To test and troubleshoot connection problems, use the script ldap-test.php located in the collector/bin folder. The script uses the same parameters as the normal collector, but produces more debug output.

So edit the configuration in the file conf/params.local.xml then launch the test script by typing the following command from the command prompt.

php collectors/bin/ldap-test.php

If you see a message like:

Error - ldap_bind('cn=admin,dc=combodo,dc=com', '*******') FAILED (Can't contact LDAP server).

then something is wrong with the connection to the LDAP server.

1. Check that parameter <ldapuri> is correct. (protocol, host and port)
2. Check that the connection to the server is not blocked by a firewall (You can use the command telnet <host> <port> and see if the connection is established).
3. Check for TSL/SSL problems. If you see the following text in the output of the ldap-test.php script, then the problem is likely related to a TLS certificate:

attempting to connect: 
connect success
TLS: peer cert untrusted or revoked (0x402)
TLS: can't connect: (unknown error code).

The solution is to instruct LDAP to ignore this faulty certificate, by adding the following lines to the LDAP configuration file (see the note below).

# Ignore the server's certificate
TLS_REQCERT never

If the output of the ldap-test.php script contains:

Error - ldap_search('dc=combodo,dc=net', '(objectClass=inetOrgPerson)') FAILED (No such object).

Then check the LDAP query used for retrieving the “contacts”. This query is defined by the two parameters:

    <ldapdn>DC=company,DC=com</ldapdn>
 
    <!-- Parameters for Person synchronization -->
    <ldappersonfilter>(objectClass=person)</ldappersonfilter>

If the LDAP query is correct, you should see an output similar to:

List of the attributes to retrieve (taken from the mapping):
uid,sn,givenname,mail,telephonenumber,mobile,title,employeenumber,memberof
Use --attributes=x,y,z to retrieve x, y and z instead. Use --attributes=* to retrieve all fields.
Debug - ldap_connect('ldaps://customers.combodo.com')...
Debug - ldap_bind('cn=admin,dc=combodo,dc=com', 'xxxxxxx')...
Debug - ldap_bind() Ok.
Debug - ldap_search('dc=combodo,dc=com', '(objectClass=inetOrgPerson)', ['uid', 'sn', 'givenname', 'mail', 'telephonenumber', 'mobile', 'title', 'employeenumber', 'memberof'])...
Debug - ldap_search() Ok.
The LDAP query '(objectClass=inetOrgPerson)' returned 13 elements.
Displaying only 10 elements (use --max-records=xx to change this limit).
------------------------------------------------
LDAP Structure:
Info: when a field is empty on a given record, it is not returned by LDAP.
------------------------------------------------
givenname : bruce
sn        : Lee
uid       : blee
mail      : bruce.lee2@combodo.com
mobile    : 0608080808
------------------------------------------------
givenname : chuck
mail      : chuck.norris@combodo.com
sn        : Norris
uid       : cnorris
------------------------------------------------

The first column of the output is the name of the field in LDAP (all fields returned by the LDAP query are listed) and the second column shows the values of the first record found in LDAP. Based on the values displayed you can complete the configuration of the mapping in the configuration file conf/params.local.xml.

Finally you can test your configuration without importing any data in iTop by running the following command from the command line:

php exec.php --console_log_level=9 --collect_only

This produces an output similar to the one shown below:

Debug - OK, the required PHP version to run this application is 5.3.0. The current PHP version is 7.2.7-0ubuntu0.18.04.2.
Debug - OK, the required extension 'simplexml' is installed (current version: 7.2.7-0ubuntu0.18.04.2 >= 0.1).
Debug - OK, the required extension 'dom' is installed (current version: 20031129 >= 1.0).
Debug - The following configuration files were loaded (in this order):

        1. /opt/dev/ldap-collector/conf/params.distrib.xml
        2. /opt/dev/ldap-collector/collectors/params.distrib.xml
        3. /opt/dev/ldap-collector/conf/params.local.xml

The resulting configuration is:

<?xml version="1.0" encoding="UTF-8"?>
<parameters>
  <itop_url>http://itop-demo/trunk</itop_url>
  <itop_login>admin</itop_login>
  <itop_password>admin</itop_password>
  <console_log_level>6</console_log_level>
  <syslog_log_level>-1</syslog_log_level>
  <max_chunk_size>1000</max_chunk_size>
  <itop_synchro_timeout>600</itop_synchro_timeout>
  <stop_on_synchro_error>no</stop_on_synchro_error>
  <curl_options>
    <CURLOPT_SSLVERSION>CURL_SSLVERSION_SSLv3</CURLOPT_SSLVERSION>
    <CURLOPT_SSL_VERIFYHOST>0</CURLOPT_SSL_VERIFYHOST>
    <CURLOPT_SSL_VERIFYPEER>1</CURLOPT_SSL_VERIFYPEER>
  </curl_options>
  <collect_person_only>no</collect_person_only>
  <ldaphost>192.168.10.13</ldaphost>
  <ldapport>389</ldapport>
  <ldapdn>OU=FGA,DC=combodo,DC=net</ldapdn>
  <ldaplogin>COMBODO\administrateur</ldaplogin>
  <ldappassword>xxxxxx</ldappassword>
  <ldappersonfilter>(objectClass=person)</ldappersonfilter>
  <itop_group_pattern>/^CN=itop-(.*),OU=.*/</itop_group_pattern>
  <person_fields>
    <primary_key>samaccountname</primary_key>
    <name>sn</name>
    <first_name>givenname</first_name>
    <email>mail</email>
    <phone>telephonenumber</phone>
    <mobile_phone>mobile</mobile_phone>
    <function>title</function>
    <employee_number>employeenumber</employee_number>
  </person_fields>
  <person_defaults>
    <org_id>Demo</org_id>
    <status>active</status>
  </person_defaults>
  <ldapuserfilter/>
  <user_id>samaccountname</user_id>
  <user_contactid>mail</user_contactid>
  <synchronize_profiles>no</synchronize_profiles>
  <user_fields>
    <primary_key>samaccountname</primary_key>
    <login>samaccountname</login>
    <contactid>mail</contactid>
  </user_fields>
  <user_defaults>
    <profile>Portal user</profile>
    <language>EN US</language>
  </user_defaults>
  <prefix/>
  <json_placeholders>
    <prefix>$prefix$</prefix>
    <persons_data_table>synchro_data_$prefix$ldap_persons</persons_data_table>
    <users_data_table>synchro_data_$prefix$ldap_users</users_data_table>
  </json_placeholders>
  <ldapfilter>(objectClass=person)</ldapfilter>
</parameters>

Debug - Persons: Mapping of the fields:
   iTop 'primary_key' is filled from LDAP 'samaccountname' 
   iTop 'name' is filled from LDAP 'sn' 
   iTop 'first_name' is filled from LDAP 'givenname' 
   iTop 'email' is filled from LDAP 'mail' 
   iTop 'phone' is filled from LDAP 'telephonenumber' 
   iTop 'mobile_phone' is filled from LDAP 'mobile' 
   iTop 'function' is filled from LDAP 'title' 
   iTop 'employee_number' is filled from LDAP 'employeenumber' 
   iTop 'org_id' is filled with the constant value 'Demo'
   iTop 'status' is filled with the constant value 'active'

Debug - LDAPUsers: Mapping of the fields:
   iTop 'primary_key' is filled from LDAP 'samaccountname' 
   iTop 'login' is filled from LDAP 'samaccountname' 
   iTop 'contactid' is filled from LDAP 'mail' 
   iTop 'language' is filled with the constant value 'EN US'
   iTop 'profile_list' is filled with the constant value 'profileid->name:Portal user'
  
...

You can see the order in which the configuration files were loaded and the resulting configuration.

If the collector does not retrieve all the expected records, this may be due to the server enforcing a limit to the length of the search results. This limit can be overcome by setting the parameter page_size (in the configuration file params.local.xml) to activate the pagination. Set a value less than the maximum number of search results allowed by the LDAP server to ensure that the data will not be truncated.

The execution of the command line will:

1. Connect to iTop to create the Synchronization Data Sources (or check their definition if they already exist, updating them if needed)
2. Connect to the LDAP server to collect the information about the Persons and the Users
3. Upload the collected data into iTop
4. Synchronize the collected data with the existing iTop Person and Users.

When the collector is run, two Synchro Data Sources are created and used for synchrponizing Person and LDAPUser objects in iTop:

Once you've run the data collector interactively, the next step is to schedule its execution so that the collection and import occurs automatically at regular intervals.

The data collector does not provide any specific scheduling mechanism, but the simple command line php exec.php can be scheduled with either cron (on Linux systems) or using the Task Scheduler on Windows.

Between version 1.1.1 and 1.2.0 the structure of the configuration has slightly changed:

• The mapping of the fields between LDAP and iTop is now defined as an array <person_fields> for the Person object and <user_fields> for the LDAPUser object. The same applies to the default values for the fields which are respectively configured in the arrays <person_defaults> and <user_defaults>. If you changed the default configuration for these items, you'll have to adjust your configuration file accordingly.
• The parameter synchro_profils has been renamed to synchro_profiles
• The parameter synchronize_organization has been deprecated. If you don't want to synchronize the organizations, don't provide a mapping for the org_id field (in <person_fields>) and provide a default value for org_id in <person_defaults>.
• Version 1.1 did not support to run several instances of the collector against the same iTop instance (for multiple LDAP servers), and thus the name of the SQL tables holding the synchro data was using a different scheme. To retain the same name of the SQL data tables, edit the configuration file and put the folling line in the <json_placeholders> section:

  <json_placeholders>
                <prefix></prefix>
                <persons_data_table>synchro_data_PersonAD</persons_data_table>
                <users_data_table></users_data_table>
        </json_placeholders>

The current version of the Data collector for LDAP supports only one source LDAP server. However you can run several instances of the collector, each with a different configuration to connect to different LDAP servers but the to the same iTop instance.

In such a configuration, make sure that the <prefix> parameter is different for each LDAP server, since each collector needs to create its own set of the Synchro Data Sources in iTop.

Create two copies of the LDAP data collector: collector-ldap1 and collector-ldap2. In collector-ldap1/conf/params.local.xml put:

<parameters>
        ...
        <ldaphost>ldap-server1.demo.com</ldaphost>
        <ldapport>389</ldapport>
        <prefix>ldap1_</prefix> <!-- IMPORTANT to have a unique prefix, use only [a-z0-9_] characters -->
</parameters>

In collector-ldap2/conf/params.local.xml put:

<parameters>
        ...
        <ldaphost>ldap-server2.demo.com</ldaphost>
        <ldapport>389</ldapport>
        <prefix>ldap2_</prefix> <!-- IMPORTANT to have a unique prefix, use only [a-z0-9_] characters -->
</parameters>

This will create two independent sets of Synchronization Data Sources:

Q: If I change the status field settings in the DataSynchro for User on iTop and the next collector run, does not restore the setting for that particular field, also it does it for others. Why?
A: LDAP does not have out of the box a simple active/unactive status for a person, so in general it is not provided by LDAP, so all User are expected to be created “active” and if no more present deleted, by the synchro. You can change that setting. One way is to change the Deletion policy, to update that status field. Another is to change the collector code source, also not sure of all the impacts of this change: