synchro_import.php

DRAFT! 🚧

Purpose

This service is used for data synchronization.

Invoke this service to import data into the data synchronization table and optionally trigger the synchronization as well.

In case you have chosen to import the data directly into the synchronization table, then you just need to trigger the synchronization. Use the service synchro_exec.php.

Arguments

Argument	Description	Defaut value
param_file	Parameters file - see Parameters file	-
auth_user	User login - CLI mode only	-
auth_pwd	User password - CLI mode only	-
data_source_id	Synchro data source id	Mandatory!
csvdata	In HTTP mode, this argument must contain the data	n/a
csvfile	In CLI mode, this arguments must point to the file containing the input data	n/a
synchronize	If set to 1, then the synchronization will be executed right after the data load	1
charset	Character set encoding of the CSV data: UTF-8, ISO-8859-1, WINDOWS-1251, WINDOWS-1252, ISO-8859-15	UTF-8
date_format	Input date format (used both for dates and datetimes) - Examples: Y-m-d, d/m/Y (Europe) Expected format - no transformation is applied if the argument is omitted
separator	column separator in CSV data (1 char, or tab)	;
qualifier	test qualifier in CSV data	“
output	`retcode` to return the count of lines in error, `summary` to return a concise report, `details` to get a detailed report (each line listed)	summary
max_chunk_size	CLI mode only, Limit on the count of records that can be loaded at once while performing the synchronization	0
~~simulate~~	~~If set to 1, then the load will not be executed, but the expected report will be produced~~	0
comment	Comment to be added into the change log
no_stop_on_import_error	Do not stop the import in case of SQL import error. By default the import will stop at the first error (and rollback all changes). If this flag is set to 1 the import will continue anyway	0

Some issues have been reported with the simulate mode. This feature remains experimental.

Excel often generates CSV using this charset=ISO-8859-1

Example

php -q /var/www/html/itop/synchro/synchro_import.php --auth_user="john" --auth_pwd="trust,no1" 
--data_source_id=2 --csvfile="mydata.csv"

Data Source

Create a Data Source for loading with:

target class = Person
reconciliation ? = email
Update = Yes for name,first_name,org_id,email
org_id (reconcialiation key) = Full name

Source file

Loading those data on an iTop with the sample data

primary_key;name;org_id;first_name;email
1;nom1;Demo;person1;person1.nom1@gmail.com
2;nom2;Demo;person2;person2.nom2@gmail.com
3;nom3;Demo;person3;person3.nom3@gmail.com
4;nom4;Demo;person4;person4.nom4@gmail.com
5;nom5;Demo;person5;person5.nom5@gmail.com
6;nom6;Demo;person6;person6.nom6@gmail.com
7;nom7;Demo;person7;person7.nom7@gmail.com
8;nom8;Demo;person8;person8.nom8@gmail.com
9;nom9;Demo;person9;person9.nom9@gmail.com
10;nom10;Demo;person10;person10.nom10@gmail.com

Pay attention to the fact that the colum header cannot be an External Field, it must be the corresponding external key (here org_id), even if the values themselves correspond to another field of the remote object (here org_id→name), that mapping is part of the 'Synchro Data Source' definition ( Reconcialiation key in Attributes tab)

There must be a primary_key column.

Normal execution

#------------------------------------------------------------
# Import phase summary
#------------------------------------------------------------
#Data Source: 2
#Synchronize: 1
#Class: Person
#Separator: ;
#Qualifier: "
#Charset Encoding:UTF-8
#Date and time format: 'Y-m-d H:i:s' 
#Date only format: 'Y-m-d' 
#Data Size: 484
#Data Lines: 10
#Columns: primary_key, name, org_id, first_name, email
#Output format: summary
#Simulate: 0
#Change tracking comment: 
#Issues (before synchro): 0
#Created (before synchro): 0
#Updated (before synchro): 10
#------------------------------------------------------------
# Synchronization phase summary
#------------------------------------------------------------
#Replicas: 10
#Replicas touched since last synchro: 10
#Objects deleted: 0
#Objects deletion errors: 0
#Objects obsoleted: 0
#Objects obsolescence errors: 0
#Objects created: 0 (0 warnings)
#Objects creation errors: 0
#Objects updated: 2 (0 warnings)
#Objects update errors: 2
#Objects reconciled (updated): 0 (0 warnings)
#Objects reconciled (unchanged): 0 (0 warnings)
#Objects reconciliation errors: 0
#Replica disappeared, no action taken: 0

2nd load of the modified source with error

Modified source file with some errors

primary_key;name;org_id;first_name;email
1;nom1;InvalidOrgName;person1;person1.nom1@gmail.com
2;nom2;2;changed-person2;person2.nom2@gmail.com
3;nom3;Demo;changed-person3;person3.nom3@gmail.com
4;nom4;IT Department;person4;person4.nom4@gmail.com
5;nom5;Demo;person5;person5.nom5@gmail.com
6;nom6;Demo;person6;person6.nom6@gmail.com

Only 6 rows are provided instead of 10
The 2 first rows contain invalid Organization full name generating: #Objects errors in update: 2
The rows 3 & 4 have been modified with allowed/valid values generating: #Objects updated: 2

Output with some Errors

#------------------------------------------------------------
# Synchronization phase summary
#------------------------------------------------------------
#Replicas: 10
#Replicas touched since last synchro: 6
#Objects deleted: 0
#Objects deletion errors: 0
#Objects obsoleted: 0
#Objects obsolescence errors: 0
#Objects created: 0 (0 warnings)
#Objects creation errors: 0
#Objects updated: 2 (0 warnings)
#Objects update errors: 2
#Objects reconciled (updated): 0 (0 warnings)
#Objects reconciled (unchanged): 0 (0 warnings)
#Objects reconciliation errors: 0
#Replica disappeared, no action taken: 0

As visible in iTop

Allowed users

Administrators will always be allowed to execute this page.

Non administrators will be allowed if they are marked as being owner of the data source.

History

Available since 1.1

Sidebar