Data Synchronization Overview

A powerful Data Synchronization engine allows multiple sources of information (for example when using a network discovery tool or an asset management tool) to be federated into iTop.

This federation is defined in iTop using Data Source Objects. See data source Each data source defines how iTop handles the synchronization of a given type of object coming from a given source.

The generic (on-going) process for synchronizing data with iTop is based on the following steps:

1. Extract data from the external source/application [not handled by iTop]
2. Transform the data to a content/format suitable for iTop [not handled by iTop]
3. Import the data into a replica table in iTop, using MySQL commands [not handled by iTop]
4. Search for matching objects in iTop [handled by iTop]
5. Create/Update or Delete the synchronized objects in iTop [handled by iTop]
6. Display and manage the synchronized object in iTop

This process is carried out in 2 possible ways :

Below is an example of the information flow from an external source into iTop, using an ETL to write the data directly in MySQL.

To implement Data Synchronization in iTop, follow the steps below:

1. From the iTop admin menu, define a Data Source object, for the type of objects you want to import/synchronize. This creates a specific replica table in the iTop database, named “synchro_data_xxx”. (Once the data source is created, check the Attributes tab of the data source. The name of the temporary table is displayed at the top.)
2. Using either your favorite ETL or plain old scripts, populate the “synchro_data_xxxx” table corresponding to the Data Source with the data coming from the external source. You can use the special column primary_key for storing any identifier of the object in the external application. Each record in this table will correspond to one object in iTop. The column id is reserved for use by iTop and cannot be written. All other columns correspond to fields of the iTop object.
3. Trigger the execution of the data synchronization between the iTop objects and the replicas of the temporary table by executing the page “synchro/synchro_exec.php” either in command line mode, or by invocating the web page (using wget for example).

Of course, you can schedule steps 2 and 3 above on a regular basis (hourly, daily or weekly).

See Replicas for more information.

An alternative method consists of creating an intermediate CSV file and using the php script synchro_import.php (provided by iTop) to perform the import into MySQL.

In this case, the information flow is as shown below:

1. Extract data from the external source/application [not handled by iTop]
2. Transform the data to a content/format suitable for iTop [not handled by iTop]
3. Write the data into an intermediate CSV file [not handled by iTop]
4. Import the data into the replica table in iTop [handled by iTop]
5. Search for matching objects in iTop [handled by iTop]
6. Create/Update or Delete the synchronized objects in iTop [handled by iTop]
7. Display and manage the synchronized object in iTop

Data Synchronization

Data Synchronization is meant to import data into iTop from another system, in a recurring manner. It can be run from the command line or from the web, but not interactively. Data Synchronization is optimized for large volumes of data that does not change often. For example, 10,000 contacts from an LDAP server, can be synchronized in iTop once a day. It is unlikely that more than a small percentage of the users' records will be modified on a daily basis. This is handled efficiently by iTop.

When synchronizing data, iTop keeps track of the relation between the iTop object and the source of the data. Therefore, it is possible to prevent users from modifying the synchronized objects (partially or totally) in iTop and to inform them where the data comes from. This is useful for “federating” several sources of data in iTop.

CSV import

The CSV import can be run interactively or from the command line. It is more targeted towards “one shot” imports. It can be used from a script (using the command line interface or from the web) or interactively. When performing a CSV import, iTop does not record information about the source of the data. Once the data has been loaded into iTop, the objects can be modified by the authorized users, without any reference to the original source.

To summarize:

CSV import is good for:

• importing initial data into iTop
• performing bulk transformations on the data (sometimes it is easier to export > modify in Excel > re-import, than to edit the objects directly in iTop)

Synchronization is good for:

• federating data between different systems in iTop
• importing data via some scheduled mechanism
• preventing users from modifying the imported data

When a configuration item is synchronized with a data source, the display is slightly different in iTop. The actions on a CI may also be limited, depending on the configuration of the data source.

For example, the fields of the object that are synchronized may appear as read-only for the end users. It is also possible to configure iTop to prevent users from deleting a synchronized object. The exact behaviour of the synchronized CI is determined by the properties of each data source.

The screenshot below shows what a synchronized Server looks like in iTop. (Notice the “lock” icon in the object's title and the tooltip attached to it).

This mechanism has been designed bearing in mind the following constraints:

• to be optimized for synchronizing huge volumes of data on a daily basis
• to integrate easily with third-party data management tools such as Talend
• to be secured and easily operated/troubleshooted

As a consequence of these requirements, setting it up is not as intuitive as one might imagine. Common pitfalls are described below:

Deleting records from the sync data table will not delete the objects. It will just detach the objects from the synchronization mechanism. Users will have the right to edit or delete the objects exactly as if they were created from within the UI. Re-attaching the objects to the synchronization requires both to setup the data source so that the reconciliation can occur, then re-introduce the records into the sync data table. See how the deletion works hereabove. The synchronization will delete the records from the synch data table for you.

With the relevant setup, this will work. However, when the table gets emptied, (and until it gets populated again AND the synchro is run repeated), users will NOT see the corresponding objects as being synchronized. Synchronization takes a long time to execute and it will heavily impact the response time on the iTop web server. For every item synchronized, a reconciliation will have to occur, requiring the execution of queries to search for the best match.

It is far easier to implement this by the means of web services such as REST/JSON. The paradigm is then more intuitive: an object is created (respectively updated/deleted) then you have to invoke the REST operation “object create” (respectively update/delete).