Data Synchronization Reference

If you are not familiar with the concept, start by the Overview

Want to see an example
Want to dig inside to troubleshoot common issues

Data Source Definition

The Data Sources are defined using the menu Admin Tools/Synchronization Data Sources.

Each data source defines:

The target class of the objects to be synchronized
The list of attributes/fields of the objects to synchronize
How to search/reconcile the objects with the objects already existing in iTop
The rules of synchronizing/updating and possibly deleting objects in iTop
How the synchronized objects behave for the iTop end-users (which fields are read-only, are users allowed to delete the objects…)
An optional hyperlink and icon to refer iTop users to the corresponding object in the external application

Description

This section of the Properties defines some general parameters for this Synchro Data Source

Property	Mandatory?	Meaning	Sample value
Name	Mandatory	The name which refers to this data source	Servers synchro from SCCS
Description	Optional	An additional description, for information only
Status	Mandatory	implementation, production or obsolete: for filtering/classifying the data sources	production
Target Class	Mandatory	The class of objects to synchronize	Server
User	Mandatory	The user allowed to execute this synchro	admin
Contact to notify	Optional	The contact to notify (by email) of the results when the synchronization is executed and contains at least one error
Icon's hyperlink	Optional	The hyperlink (URL) to an image representing this data source. This icon is shown instead of the default icon in the tooltip appearing on the “Lock” symbol next to the title of the synchronized object in iTop
Application's hyperlink	Optional	The hyperlink (URL) to the remote application from which the synchronized data comes from. The hyperlink can contain placeholders such as $this->attcode$ (which will be replaced by the value of the attribute of the current object) or $replica->column_name$ (which will be replaced by the value of the named column from the replica)	https://myapp.acme.com/display/server/$replica->primary_key$
Data table	Optional	The name of the MySQL table in which the replica information will be stored. If omitted, the system will generate a unique name for you.	synchro_data_servers_from_sccs

The table which will hold the replicas, is created based on the class datamodel, if the datamodel is changed later this table will be updated by the setup (as usual column will be added but not removed).
Because it was required to be able through synchro_import.php to empty fields such as date, the format of date column was changed in release 2.3.4 from date to varchar(10)

Search & reconciliation

This section of the properties defines

How the information contained in the replica table is matched against the content of the iTop database
What is the behavior of this synchro data source concerning the creation of new iTop objects.

Property	Meaning
Reconciliation policy	`Use the attributes` (use_attributes): when a new replica is created, the system will search for a matching object in iTop based on the list of Synchro attributes with the flag `Reconciliation` set to `Yes`. `Use the primary key` (use_primary_key): the column `primary_key` of the replica is expected to contain the identifier of the object in iTop.
Action on zero	What to do when the reconciliation did not find any object in iTop. `Create` instructs the system to create a new object of the Target Class. `Error` will cause the system to report this case as an error (i.e. this synchro data source is supposed to only update existing objects).
Action on one	What to do when the reconciliation found exactly one matching object in iTop. `Update` instructs the system to update this object. `Error` will cause the system to report this case as an error (i.e. this synchro data source is supposed to create objects and never update existing objects).
Action on many	What to do when the reconciliation found several matching objects in iTop. `Create` instructs the system to create another object. `Error` will cause the system to report this case as an error. `Take the first one` instructs the system to pick the first object in the list and to update it.

Deletion rules

This section defines how the deletion of the synchronized objects are handled:

by the end-users in iTop,
by the synchronization system itself.

Property	Meaning
Users Allowed	Which end-users are allowed to delete synchronized objects in iTop. `Nobody` when an object is synchronized, nobody is allowed to delete it in iTop. only the synchronization system can delete it. `Administrators only` only administrators are allowed to delete a synchronized object. `Everybody allowed to delete such objects` the normal access rules apply to such synchronized objects.
Full load interval	The duration between two executions of the synchronization which are considered as having loaded (touched) all replicas. When a replica has not been touched for a duration greater than this value, it is considered as obsolete.
Delete Policy	What to do when a replica becomes obsolete. `Ignore` do nothing, the associated object remains as is in iTop. `Delete` Delete the associated object in iTop (and the replica in the data table). `Update` Update the associated object as specified by the `Update rules` (see below). `Update then Delete` Update the associated object as specified by the `Update rules` (see below), then after the `Retention Duration`, delete the object (and the replica).
Update rules	A list of values separated by semi-colons, of the form `attribute_code:value` to specify which attribute of the associated object to set and to which value. Example: `status:obsolete;description:no longer synchronized`.
Retention duration	Only used when the `Delete Policy` is set to `Update then delete`. In that case, an object deleted from the Source is updated immediately and deleted from iTop only after that `Retention duration`.

If you set Full load interval = 0s and run the synchro with a collector, then deletion policy will do nothing

This parameter Full load interval is ignored by a synchro_import.php --synchronize=1 as any object not provided within that unique import is considered as 'obsolete'

If your set a Full load interval of 7 days, and a Retention duration of 14 days, an object no more provided by the source will be deleted after 3 weeks.

Attributes

The “Attributes” tab defines:

How to determine whether a new object must be created in iTop or an existing object must be associated with the replica (if the reconciliation rule is “Use Attributes” in the properties)
Which attribute(s) of the associated object must be updated, and how.

Also it is possible to update an AttributeLinkedSetIndirect within a DataSynchro, this method cannot be used with a large number of relations (limitation of the column size within the replica table) and should not be used with an often changing relationship for performance reason.
In those case, use a separated DataSynchro, to synchronize the n:n relationship itself

Attributes tab	Purpose
Attribute	It automatically lists each attributes of the DataSynchro “Target class”, with its label and code
Reconciliation	Specify if the value in this column should be used to retrieve an existing iTop object. Used for new replica only
Update	Specify if that iTop object attribute should be updated by the Datasynchro
Update Policy	Specify if the attribute can be edited in iTop or not
Reconciliation Key	Required for ExternalKey attributes only. Specify one attribute of the remote object used to retrieve it. It can be the id or a field which would be a unique identifier for the remote class

The Reconciliation column

When a new record appears in the synchro_data_xxx table, the synchronization mechanism will try to associate this record with an existing iTop object. If one or more Attribute is marked as “Yes” in the “Reconciliation ?” column, these columns/attributes will be used to search for an iTop object that matches all these values. If one object is found, it will become associated with this record. If no object is found, a new iTop object will be created.

The definition of appropriate reconciliation criteria is the key to the proper behavior of the data synchronization. In order for the synchronization to behave properly, you must use a list of attributes that uniquely identify objects to synchronize in a reliable manner, as reconciliation criteria.

The Update column

The Update column defines if the attribute must be updated or not.

Checked = Yes: the attribute will be set/updated
Unchecked = No: the external source information for this attribute will be ignored, the existing value on the iTop object will be kept as is.

When updating iTop object, Null and empty string are not equivalent

Null ⇒ The existing iTop field value is kept as is,
Empty string ⇒ Reset iTop object field

You must use your own feeding mechanism for loading Null values in synchro_data_xxx table, as synchro_import.php can't do it

Limitations: Some field type cannot be reset by DataSynchro: Enum, Integer and Decimal

The Update Policy column

The column Update Policy defines the behavior of the updated attribute when editing the iTop object. The possible values are:

Locked: the attribute will be completely driven by the data synchronization and will appear as read-only (with a small lock icon next to it) in the iTop user interface,
Unlocked: the attribute will be updated by the synchronization (whenever the source data changes) but end-users can still modify the attribute in parallel inside iTop,
Initialize if empty: the synchronization will only set a value of this attribute if the value is empty in iTop. The attribute remains modifiable inside iTop.

The Reconcialiation key column

Finally, the Reconciliation key column defines how to process attributes which are references to another iTop object (i.e. Foreign keys). The supplied data can contain either the actual value of the foreign key (id) or another attribute such as the name of the object; which must uniquely identify the object.

For foreign keys, iTop does not support the reconciliation on an combination of the multiple attributes (like name AND organization). If such a need arises, then you must use an ETL or a script to perform the desired reconciliation before providing the id to iTop.

Status

The Status tab displays information about the execution of the synchronization. If the synchronization has never been run, the tab displays the following message:

This synchro was never run. No log yet.

Otherwise the tab displays a schema similar to the one below:

Synchronization history

On the left of the screen, each execution of the synchro is displayed in an historical list, showing the overall status, the start date and time of the execution.

Click on an element of this list to show the corresponding status in detail, displayed in the two columns on the right.

The column at the left displays the status for the records in the synchro_data_xxxx table:

New records are records which were added in the table since the previous execution of the synchro,
Existing records are records which were existing before and have been loaded again since the previous execution of the synchro,
Disappeared records are records which were not loaded since more than the full_load_interval duration.
Ignored records corresponds to objects no longer existing in iTop.

The column at the right displays the consequences of the synchro, in terms of modifications on the associated objects in iTop.

For each New record, the consequences can be either:
- The creation of the new object in iTop (Created),
- The update of an already existing object in iTop (Updated),
- The simple association with an object in iTop, which is already up-to-date (Unchanged),
- Or an Error when trying to either create or update an object in iTop (most probably because of inconsistent data).

For each Existing record, the consequences can be either:
- The update of the associated object (Updated),
- No change at all (Unchanged)
- Or an error when trying to update the associated object (Errors)

For each Disappeared record, the consequences (depending on the “Deletion rules”) can be either:
- The associated object in iTop has been deleted (Deleted),
- The associated object in iTop has been updated (Updated),
- The associated object in iTop was left unchanged (No Action),
- Or an error occurred when trying to update or delete the associated object in iTop (Errors).

Replicas

In iTop, the Replicas are used to store the status information and the relation between the raw data loaded in the synchro_data_xxx table and the associated iTop object.

The Replicas for a given data synchronization source can be displayed by clicking on the links at the top of the status tab. A Replica is displayed as shown below:

The content of the replicas is useful to understand the behavior of the data synchronization mechanism (and to get more information about the errors), but its data are maintained by the synchronization and should never be modified manually.

Fields of a Replica	Purpose
Synchro Data Source	Synchro Data Source to which this Replica belongs
Destination object (ID)	iTop object id corresponding to that replica. Can be empty when the replica is new and could not be synchronized due to an error
Destination type	The Target class of the Synchro Data Source, which is also the class of the Destination object
Status	Status of the Replica: Modified, New, Obsolete, Orphan, Synchronized
Last seen	When that replica was seen for the last time in the source data (same `primary_key`)
Object created?	Was the iTop object created by the Data Synchro?
Last Error	Error
Warnings	Warning message when some non blocking error occured
Creation date	When the iTop object was created by the Synchro Data Data
Last Modified Date	When that replica was modified by the source for the last time

Replica's status	Meaning
New	Newly received replica. Temporary status before syncho_exec.php occurs to create the iTop object
Modified	Modified replica. Temporary status before syncho_exec.php occurs to update the iTop object
Synchronized	Replica and iTop object are aligned
Obsolete	Data is no more in the Source, iTop object was not deleted, otherwise the replica would be as well
Orphan	This should not happen. In this situation the synchronized iTop object has disappeared without deleting the replica at the same time. Such replica can safely be manually deleted

Using MySQL statements to populate the synchro_data_xxxx table

If you populate the table synchro_data_xxx using SQL commands (either with an ETL or a home made script), you must take care of never creating duplicate records. Remember that each record in the synchro_data_xxx table corresponds to exactly one object in iTop.

In order to avoid such duplicates, you must first query for an existing record before inserting a new record into the table. To speedup this query you can either construct a value that uniquely identifies the “source” object and store this value in the “primary_key” column of the table (which has a unique index), or use any combination of the record's column and alter the definition of the synchro_data_xxxx table to add your own indexes.

Do not use the MySQL statement INSERT … ON DUPLICATE KEY UPDATE to insert records in the synchro_data_xxx table. This is not supported by the internal iTop mechanism which relies on the “BEFORE INSERT” trigger.

Should you need to empty the synchro_data_xxxx table, do not use the MySQL statement TRUNCATE TABLE, since this statement will actually drop and reconstruct the table… but not its triggers! Instead use

DELETE FROM
synchro_data_xxxx

.

Then trigger the execution of the data synchronization between the iTop objects and the replicas of the temporary table by executing the page “synchro/synchro_exec.php” either in command line mode, or by invocating the web page (using wget for example).

Using synchro_import to populate the synchro_data_xxxx table

See: synchro_import.php

The synchronize parameter of synchro_import.php trigger synchro_exec.php by default

How to specify a list of related objects (link set)

Some classes have attributes called Link Sets. Example: UserLocal.profile_list

You can synchronize those lists by writing all the information in the corresponding column:

Here is an example specifying that the login will have one profile: 'Portal User'

INSERT login, profile_list INTO data_synchro_userlocal_1 VALUES ('johndoe', 'profileid->name:Portal User;reason:Customer')...;

You can also specify several links (several profiles).

Troubleshooting

Data Synchro Troubleshooting

Sidebar