Data collector for Graph

This collector enables administrators to automatically feed iTop with relevant and accurate Graph cloud computing information.

The Data collector for Graph is a stand-alone PHP software that connects to Microsoft Graph, retrieves objects from that environment and synchronizes them with iTop's CMDB.

• Connection to Graph is made through Graph OAUTH2 authentication,
• Retrieval of objects relies on Graph REST API that extracts data in JSON format,
• Synchronization follows iTop's built-in Data Synchronization mechanism.

The extension focuses on the following Graph objects that can be filtered before being synchronized within iTop:

• Groups
• Members
• Users

They will be mapped to the following iTop classes:

• Team
• link Person / Team
• Person and User

Data collection can be enhanced by adding further Graph classes to this list. Please, refer to the Q&A chapter below.

The collector only synchronizes a limited subset of objects from Graph (see above). Neither the Organizations nor the Locations are synchronized.

By default, the collector considers that iTop persons are reconciled by their email attribute.

Usage of the collector requires you to comply with a few points:

• The Graph objects synchronized by the collector must exist in iTop (Users or CMDB CIs). This is, by default, the case with iTop's standard data model.
• In order to browse Graph's computing environment, you'll need a Graph account (see service principal) that allows you to read Graph's resources.
• From a system standpoint, you'll need to comply with the requirements expressed in the Data collector Base documentation.

Simply expand the content of the zip archive into a folder on the machine where the collector will be run.

The configuration of the application is built by concatenating 4 files:

1. <graph_collector>/collectors/params.distrib.xml that holds entries that are specific to the Data collector for Graph. It should not be modified.
2. <graph_collector>/collectors/extensions/params.distrib.xml that holds entries that have been created and added by the customer to answer its specific needs. This file is optional.
3. <graph_collector>/conf/params.distrib.xml that is provided by the collector framework, Data collector Base. It should not be modified.
4. <graph_collector>/conf/params.local.xml where the collector can be adapted to the specific customer needs.

The conf/params.distrib.xml configuration file lists, amongst others, the parameters that are required for the collector to connect to iTop. Proper ones must be changed through the conf/params.local.xml file.

<?xml version="1.0" encoding="UTF-8"?>
<!-- Default values for parameters. Do NOT alter this file, use params.local.xml instead -->
<parameters>
  <itop_url>https://localhost/iTop</itop_url>
  <itop_login>admin</itop_login>
  <itop_password>admin</itop_password>
  <itop_token/>
  <itop_login_mode/>
 
  ... Other general parameters ...
 
</parameters>

The collectors/params.distrib.xml configuration file holds parameters that must (for some) or can (for others) be changed when configuring the collector, which must be done through the conf/params.local.xml file.

<?xml version="1.0" encoding="UTF-8"?><!-- Default values for parameters. Do NOT alter this file, use params.local.xml instead -->
<parameters>
        <!-- Microsoft links -->
        <microsoft_login_url>https://login.microsoftonline.com/</microsoft_login_url>
        <microsoft_auth_mode>/oauth2/token</microsoft_auth_mode>
        <microsoft_resource>https://graph.microsoft.com/</microsoft_resource>
 
        <!-- Client's credentials -->
        <ms_clientid></ms_clientid>
        <ms_clientsecret></ms_clientsecret>
        <ms_tenantid></ms_tenantid>
        <ms_tenantid_short></ms_tenantid_short>
 
        <!-- *** Graph Class Parameters *** -->
        ...
           List of Graph classes with their parameters
        ...
 
        <!-- Class collection sequence -->
        ...
           List of classes to collect with their rank in the collection process
        ...
 
        <!-- Synchro data source parameters -->
        <contact_to_notify></contact_to_notify>
        <synchro_user>admin</synchro_user>
        <name_prefix>Graph</name_prefix>
        <name_postfix>Discovery</name_postfix>
        <json_placeholders>
                <itoplnkpersontoteam_synchro_name>$name_prefix$ lnkPersonToTeam $name_postfix$</itoplnkpersontoteam_synchro_name>
                ...
                   Placeholders for other classes
                ...
                <synchro_status>production</synchro_status>
                <full_load_interval>604800</full_load_interval><!-- 7 days (in seconds): 7*24*60*60 -->
                <users_target_class>UserExternal</users_target_class>
                <!-- Set the following parameters to 1 if you want to update them, to 0 otherwise -->
                <update_persons_employee_number>1</update_persons_employee_number>
                <update_persons_manager>1</update_persons_manager>
                <update_persons_organization>1</update_persons_organization>
                <!-- Following policies may be: write_if_empty, master_locked or master_unlocked
                 . Use write_if_empty if you want the collector to initialize the profiles for each account but let you edit them in iTop after the initial creation of the user
                 . Use master_locked if you manage the profiles directly in Graph and let the collector synchronize them in iTop
                 . Use master_unlock  if you don't want the list of profiles to be synchronized
                 -->
                <profile_list_update_policy>master_locked</profile_list_update_policy>
        </json_placeholders>
 
        <!-- mapping tables -->
        <user_language_mapping type="array">
                <pattern>//EN US</pattern>
                <pattern>/en/EN US</pattern>
                <pattern>/en-US/EN US</pattern>
                <pattern>/.*/%1$s</pattern>
        </user_language_mapping>
</parameters>        

The collectors/extensions/params.distrib.xml configuration file holds parameters to collect additional classes required by a customer.

<?xml version="1.0" encoding="UTF-8"?><!-- Default values for parameters. Do NOT alter this file, use params.local.xml instead -->
<parameters>
        <!-- Graph Class Parameters -->
        ...
           List of additional Graph classes with their parameters
        ...
 
        <!-- Class collection sequence -->
        ...
           List of additional classes to collect with the rank in the collection process
        ...
 
 
        <!-- Synchronization parameters -->
        <json_placeholders>
                <graphmyclass_synchro_name>$name_prefix$ My Class $name_postfix$</graphmyclass_synchro_name>
                ...
                   Placeholders for other additional classes
                ...
        </json_placeholders>
 
</parameters>

This set of parameters is required to connect to iTop application or to Microsoft Azure environment. Some of them must or may be adjusted to meet customers' own environment.

The parameters that are defined in this section will directly alter the content of the json files used by the collector to build the synchronization data sources in the remote iTop. Some of them must or may be adjusted to meet customers' environments.

This section groups the mapping tables used by the collector.

Next to the core parameters described here above, the collectors/params.distrib.xml file provides the list of all Graph classes that need to be discovered. For each of them, it lists the parameters that should be synchronized within iTop and the filters that may be applied on the extracted data. Since the retrieval of Graph data is made in JSON format, this list must be aligned with the requirements expected by the standard JSON collector defined in the data collector base.

General format is as follows:

<itopNAME_OF_THE_CLASSgraphcollector>
        <ms_class></ms_class>
        <api_version></api_version>
        <jsonfile></jsonfile>
        <path></path>
        <fields>
                <primary_key>id</primary_key>
                <itop_attribute#1>graph_attribute#1</itop_attribute#1>
                <itop_attribute#2>graph_attribute#2</itop_attribute#2>
                ...
                <itop_attribute#n>graph_attribute#n</itop_attribute#n>
        </fields>
        <group_filters>
                <prefixes>
                        <graph_attribute>prefix_value</graph_attribute>
                        or
                        <graph_attribute type="array">
                                <prefix>prefix_value#1</prefix>
                                <prefix>prefix_value#2</prefix>
                        </graph_attribute>
 
                </prefixes>
        </group_filters>
        <defaults>
                <status>active</status>
                <org_id>Demo</org_id>
        </defaults>
        <nullified_attributes type="array">
                <attribute>itop_attribute#n</attribute>
        </nullified_attributes>
</itopNAME_OF_THE_CLASScollector>

The filters may be removed or changed to fit the real configuration of the Graph application.

This section defines the list of classes that will be collected and in which order. It enables as well the possibility to deactivate the collection of a class.

<collectors_launch_sequence type="array">
        <!-- iTop Teams -->
        <collector>
                <name>iTopTeamGraphCollector</name>
                <enable>yes</enable>
                <rank>1</rank>
        </collector>
        <!-- iTop lnkPersonToTeamGraphCollector -->
        <collector>
                <name>iToplnkPersonToTeamGraphCollector</name>
                <enable>yes</enable>
                <rank>2</rank>
        </collector>
        <!-- iTop Persons -->
        <collector>
                <name>iTopPersonGraphCollector</name>
                <enable>yes</enable>
                <rank>3</rank>
        </collector>
        <!-- iTop Users -->
        <collector>
                <name>iTopUserGraphCollector</name>
                <enable>yes</enable>
                <rank>4</rank>
        </collector>
</collectors_launch_sequence>

The launch of the Graph collector will be driven by the command and parameters defined in the usage section of iTop Data collector base. Once launched, first action of the collector will be to build its collection plan, based on the list of classes that have been enabled in the configuration file. Then,

• Configuration files will be consolidated,
• Synchronisation data sources will be created or updated if required,
• Authentication to Graph environment will be made (see below),
• Collection of Graph classes will be made by invoking Graph REST API and extracted data will be stored under the local “data” directory, in JSON format,
• Synchronisation will run and Graph objects will be pushed to iTop.

This must be the first class to be collected. Defaults parameters for that class are:

<itopteamgraphcollector>
        <ms_class>Group</ms_class>
        <api_version>1.0</api_version>
        <jsonfile>data/iTopTeam.json</jsonfile>
        <path>value</path>
        <fields>
                <primary_key>id</primary_key>
                <name>displayName</name>
                <org_id>org_id</org_id>
                <email>mail</email>
                <status>status</status>
        </fields>
        <group_filters>
                <prefixes>
                        <displayName type="array">
                                <prefix>G_ITOP_TEAM_</prefix>
                                <prefix>G_ITOP_PERSON_</prefix>
                                <prefix>G_ITOP_PROFILE_</prefix>
                        </displayName>
                </prefixes>
        </group_filters>
        <defaults>
                <org_id>Demo</org_id>
                <status>active</status>
        </defaults>
</itopteamgraphcollector>

All Graph groups are collected but the defined filters make sure that the only groups which displayName is prefixed wih G_ITOP_TEAM_, G_ITOP_PERSON_ or G_ITOP_PROFILE_ are kept and processed by the further steps of the collector.

The Graph groups to consider being identified at this stage, their members can be retreived. The lnkPersonToTeam iTop class is the one that links persons with teams in a n:n relation. It must be the second one to be collected, with the following defaults parameters:

<itoplnkpersontoteamgraphcollector>
        <ms_class>Member</ms_class>
        <api_version>1.0</api_version>
        <jsonfile>data/GraphMember.json</jsonfile>
        <path>value</path>
        <fields>
                <primary_key>primary_key</primary_key>
                <person_id>graphperson_id</person_id>
                <team_id>graphgroup_id</team_id>
        </fields>
        <defaults/>
</itoplnkpersontoteamgraphcollector>

Once and only once the Team and lnkPersonToTeam classes have been collected, the Person class is collected as well. Defaults parameters for that class are:

<itoppersongraphcollector>
        <ms_class>User</ms_class>
        <api_version>1.0</api_version>
        <jsonfile>data/GraphPerson.json</jsonfile>
        <path>value</path>
        <fields>
                <primary_key>id</primary_key>
                <email>mail</email>
                <employee_number>employeeId</employee_number>
                <first_name>givenName</first_name>
                <function>jobTitle</function>
                <location_id>officeLocation</location_id>
                <manager_id>manager_id</manager_id>
                <mobile_phone>mobilePhone</mobile_phone>
                <name>surname</name>
                <org_id>org_id</org_id><!--org_id>companyName</org_id-->
                <phone>businessPhones</phone>
                <status>status</status>
        </fields>
        <group_filters>
                <prefixes>
                        <displayName type="array">
                                <prefix>G_ITOP_PERSON_</prefix>
                        </displayName>
                </prefixes>
        </group_filters>
        <defaults>
                <status>active</status>
                <org_id>Demo</org_id>
        </defaults>
        <nullified_attributes type="array">
                <attribute>manager_id</attribute>
        </nullified_attributes>
</itoppersongraphcollector>

Version 1.0 of the Graph API does not allow us to build a request that filters users according to the group they belong to (G_ITOP_PERSON_ for instance), with, at the same time, a specific set of attributes, like the ones given in the <fields> section above, and information related to their manager. As a consequence, all Graph users are collected but only the ones that belong to the groups that match any of the filters are kept before being processed by the further steps of the collector.

This is the last class to be collected. Defaults parameters are:

<itopusergraphcollector>
        <ms_class>User</ms_class>
        <api_version>1.0</api_version>
        <jsonfile>data/GraphUser.json</jsonfile>
        <path>value</path>
        <fields>
                <primary_key>userPrincipalName</primary_key>
                <contactid>mail</contactid>
                <language>preferredLanguage</language>
                <login>userPrincipalName</login>
                <profile_list>profile_list</profile_list>
                <status>status</status>
        </fields>
        <group_filters>
                <prefixes>
                        <displayName type="array">
                                <prefix>G_ITOP_PROFILE_</prefix>
                        </displayName>
                </prefixes>
        </group_filters>
        <defaults>
                <status>enabled</status>
        </defaults>
        <nullified_attributes type="array">
                <attribute>contactid</attribute>
        </nullified_attributes>
</itopusergraphcollector>

Like for the Person class, all Graph users are collected but only the ones that belong to the groups that match any of the specified filters are kept before being processed by the further steps of the collector.

A user that is attached to several G_ITOP_PROFILE_ groups in Graph will be created / synchronized in iTop with all the profiles corresponding to these groups. For instance:

• To create an iTop user with the Portal user profile only, attach the corresponding Graph user to the G_ITOP_PROFILE_Portal group only.
• To create an iTop user with both the Configuration Manager and the Service Manager profiles, attach the corresponding Graph user to both the G_ITOP_PROFILE_Configuration Manager and the G_ITOP_PROFILE_Service Manager groups.

Question: how does the authentication mechanism work ?
Answer: the collector relies on OAUTH2 authentication requested by Graph to retrieve data through its REST API. This is the exact same mechanism as the one implemented within the Data collector for Azure. Please, refer to the Q&A section of that collector for further details.

Question: how can I add Graph classes to the collection plan ?
Answer: Other classes than the ones listed by default in the collector can be added to its collection plan. For that purpose, we highly recommend to use the extensibility mechanism that is provided by the Data Collector base. You'll find the related documentation here.

Question: how can I authenticate iTop users with Graph collected information ?
Answer: Special care must be taken when iTop users rely on Graph to authenticated themselves on iTop. Let's explain why.

Such authentication requires iTop to communicate with Microsoft Graph through oauth2 protocol to verify the credentials of a user. This protocol, and the way it is implemented within iTop, behaves as follows:

1. The Graph attributes of the user who is requesting acess to iTop are, first, read from Graph,
2. Then, the mail Graph attribute is checked:
   1. If this attribute exists and is not empty, it will be considered by the protocol as the login of the user.
   2. If this attribute does not exist or is empty, the protocol will consider the userPrincipalName attribute as the login for the user

Consequence for the configuration of the Data Collector for Graph, namely for the iTop class Person:

• If the email of a person (mail attribute in Graph) and the login for the corresponding user (userPrincipalName in Graph) are the same, then the default configuration for the class Person can be used:
  • <email>mail</email>
• If email and login are different, i.e. for a given person, the email used for notification is not used as a login,
  • then the Graph configuration should be set as follows:
    • mail attribute should remain empty or be the same as the userPrincipalName one
    • the person's email should be stored in first raw of the Graph array attribute otherMails
  • and the collector configuration for the class Person should be:
    • <email>otherMails</email>

Question: how can I synchronize different tenants in the same iTop ?
Answer: The simplest way to do so is to deploy and run different collectors, one per tenant.