This extension runs in the background to scan the defined mail inbox(es) and either create or update tickets based on the content of the incoming emails.

• On Ticket creation, it fills the description of the ticket with the content of the email, set the caller, the customer, copy attachments, add contacts and many other fields…
• On Ticket update, it extracts as best as possible, the last reply part of the email to update the Public Log of the ticket, copy attachment, change the status of the ticket and add contacts

• Determine if the sender is an existing Person (found by its email), then based on configuration it can reject the email in error or create a new Person.
• Determines whether a Ticket must be created or updated based either on the custom headers added by iTop in the eMail (in case of replies) or based on a configurable pattern in the title
• Connect to any mailbox using either the POP3 or IMAP protocol
• Interactive configuration of the mail inboxes
• Processes the incoming eMails in both HTML or plain text format
• Support both Incidents and User Request tickets
• Ticket's attachments are automatically turned into attachments of the Ticket (“dangerous” types of attachments can be excluded)
• Automatic detection of duplicate attachments (like signature images)
• Keeps the messages in the mailbox until the corresponding Ticket is either closed or deleted
• Manual retry in case an error occurs when processing an eMail
• Images embedded inside an HTML mail are displayed inline in iTop as well.
• Images “too small” (below configurable dimensions) are not imported as attachments (to exclude signatures)
• Images bigger than configurable dimensions can be resized automatically before uploading into iTop
• “Inline images” are displayed at a reduced (configurable) size and can be “zoomed-in” by clicking on them.
• Automatically reject “Autoreply” emails, based on a set of configurable patterns to be tested against the subject of the email
• Add the other recipients of the email (To: and CC:) as additional contacts on the ticket (configurable). The contact must already exist with that exact email, it won't be created, just linked to the Ticket..
• Apply a stimulus (configurable, depending on the state of the ticket) to change the state of the ticket upon reception of an email.
• Add a new type of Trigger: Trigger (when updated by mail) which allow to notify when a Ticket is updated through a received email.
• New in 3.4.1: Add 2 delegation profiles: Mail Inbox Manager to manage mailboxes and Mail Messages Manager to manage the messages

• PHP 7.1 is required for accessing a Google Gmail or Microsoft Azure mailbox. In addition, you will need to create an OAuth client.
• PHP 5.2.1+ with the IMAP extension enabled if you want to connect to an IMAP server,
• A connection to a IMAP server with a valid mailbox.
• cron.php must be running to enable the processing of incoming eMails.
• For the debug trace, PHP MBString must be installed.
• For resizing big images, PHP GD must be installed.

.

• Each mail inbox configured corresponds to one type of Ticket (i.e. User Requests or Incidents, but not both).
• Connecting to a POP3 mailbox via the php IMAP extension does not work. You must use the POP3 configuration via PEAR::NetSocket for such a case.
• There is no validation that the target class of the Mail Inbox is an existing class in iTop. (i.e. don't try to create incidents if the Incident Management module is not installed).
• The support of inline images works only if the preferred order for email parts is configured as “text/html,text/plain” (i.e. HTML first), which is now the default for all new installations.

Use the Standard installation process for this extension.

When creating a new ticket from an incoming email message, the application automatically fills the following fields of the ticket:

• Title = subject of the mail
• Description = body of the mail
• Caller (caller_id) = sender of the mail (identified by her/his email address)
• Organization (org_id) = organization of the caller
• Origin (if this field exists on the ticket) = 'mail'

Default (i.e. constant) values must be supplied, via the “Ticket Default Values” setting, for any other mandatory field of the Ticket, otherwise the creation of the Ticket will fail.

When updating an existing Ticket, the application add an entry into the public_log field (configurable, see below), with the “new part” of the message. See below for more explanation about how the “new part” is extracted from the message.

The syntax for specifying default values is the following:

• One field per line, with the syntax:
• field_code:default_value

If the field to initialize is a key to another object (for the fields like org_id, service_id, etc…) you can specify either the numeric identifier of the target object (e.g. service_id:153) or its name (e.g. org_id:Demo), provided that the name is unique.

The behavior for each mailbox (which messages to process and whether to create or update a Ticket) is managed via the menu “Incoming eMail Inboxes” in the “Admin tools” section:

Click, “Create a new Mail Inbox” to create a new configuration for a mail inbox. This displays the following form:

From iTop 2.7.7, you get another type of Mailbox to connect to Gmail and Azure using OAuth2

The “Mailbox configuration” defines how the application connects to the mail inbox:

This section defines the behavior when an incoming email cannot be processed properly. The email can be either kept in the mailbox (and remembered as an “Error” and no longer processed) or deleted immediately from the mailbox. Furthermore the original message can be forwarded to an administrator (as an attachment) along with some explanation about the cause of the error.

This section defines the behavior of the application when processing incoming emails.

This section determines the behavior of the application when the sender of an email (From:) does not correspond to a known email address in the application. There are two possibilites:

• Reject the eMail: the incoming email is treated as an error and thus either forwarded to an administrator or deleted.
• Create a new person: a new Person will be created based on the email of the sender and constant values defined below.

This section determines the behavior of the application regarding the additional recipients of the incoming email (persons in To: and CC: of the message). It is possible to specify if/when the email addresses which correspond to a valid contact in iTop are added to the ticket (via the Contacts tab). Email addresses which do not correspond to a valid contact in iTop are always ignored.

In addition to the configuration performed by creating a Mail Inbox object using the user interface of the application, a few parameters are available in the configuration file to fine tune the behavior of the application.

Configuration file

   'combodo-email-synchro' => array (
      'debug' => false,
      'periodicity' => 30,
      'body_parts_order' => 'text/html,text/plain',
      'pop3_auth_option' => 'USER',
      'imap_options' => array (
            0 => 'imap',
            1 => 'ssl',     // Mandatory for OAuth2
      ),
      'imap_open_options' => array(),
      'exclude_attachment_types' => array (
            0 => 'application/exe',
      ),
      'maximum_email_size' => '10M',
      'recommended_max_allowed_packet' => 10485760,
      'introductory-patterns' => array (
            0 => '/^le .+ a écrit :$/i',
            1 => '/^on .+ wrote:$/i',
            2 => '|^[0-9]{4}/[0-9]{1,2}/[0-9]{1,2} .+:$|',
      ),
      'multiline-delimiter-patterns' => array (
            0 => '/\\RFrom: .+\\RSent: .+\\R/m',
            1 => '/\\R_+\\R/m',
            2 => '/\\RDe : .+\\R\\R?Envoyé : /m',
            3 => '/\\RDe : .+\\RDate d\'envoi : .+\\R/m',
            4 => '/\\R-----Message d\'origine-----\\R/m',
            5 => '/\\RExpéditeur: .+\\RDate:/m',
            6 => '/\\RDe : .+\\RDate : /m',
            7 => '/\\TO:.+\\RCC:/m',
      ),
      'delimiter-patterns' => array (
            '/^>.*$/' => false,  // "false" remove only the line, "true" remove the rest of the message
      ),
      'big_files_dir' => '',
      'use_message_id_as_uid' => false, // Don't change this unless you know what you are doing!
      'images_minimum_size' => '100x20',
      'images_maximum_size' => '',
      'undesired-subject-patterns' => array (
            0 => '/^Out Of Office/i',
            1 => '/^Automatic answer$/i',
            2 => '/^Réponse automatique:/',
      ),
      'undesired-purge-delay' => 7,
      'html-tags-to-remove' => array(
           'blockquote' => array(), 
           // No class specified, remove any blockquote tag
           'div' => array('gmail_quote', 'moz-cite-prefix'),
           'pre' => array('moz-signature'),
      'retention_period' => 1, // number of hours we keep the replicas
      );
 
   ),

        'itop-standard-email-synchro' => array (
                'ticket_log' => array (
                          'UserRequest' => 'public_log',
                          'Incident' => 'public_log',
                        ),
                 'aggregate_replies' => true // same as field not present       
        ),

Once the Mail Inbox object has been created, you can use the tab “Mailbox Content” in the details of the object to check that the application can properly connect to the mail server and retrieve messages from it.

This view also allows to perform two different kind of actions on a set of messages:

• “Reset Status”: for messages which are either flagged as “Error” or “Already processed”, this status will be reset and the email will considered again as “New” and thus candidate for processing the next time cron.php runs.
• “Delete eMail”: deletes the message(s) from the mailbox. No confirmation will be asked !!
• “Ignore eMail”: mark new messages as “ignored” to avoid processing them.

New in 3.0.15: Access to the eml used and to the message specific logs without activating the debug mode for every processed message.

Since the processing of the incoming emails occurs in the background, it is not always easy to understand what happens when a ticket is not processed as expected. In order to trace the execution of this background task, several levels of tracing are available:

• by setting the “Debug trace” field to “Yes”, more trace will be generated when processing the mailbox. This trace goes to the output of the cron job, and is also captured (limited to 256 KB) in the database. This captured trace is visible in the “Debug Trace” tab of the Mail Inbox object.
• the configuration setting debug for the module combodo-email-synchro can be set to true in the configuration file. This activates even more traces for all the configured mail inboxes. This additional trace appears only in the output of the cron job.
• the cron job can be passed the optional parameter --verbose=1 to activate some debug trace for all the background tasks. This additional trace also goes into the output of the cron job.

If the cron job is not already running in the background it is convenient to run it manually from the command line, to see what's happening:

  php cron.php --auth_user=<user> --auth_pwd=<pwd> --verbose=1

Question: Mails are no more processed?
Answer:

1. It might be due to a email message with a too big attachment, which can end-up crashing the database. In that case, to quickly process the following emails, open the Mail Inbox in iTop, tab: Mailbox Content; identify the faulty email, probably the oldest “new” message and check it and press the Ignore button.
2. To limit the above situation, check coherence between allowed_max_packet, recommended_max_allowed_packet and maximum_email_size parameters.
3. It might be due to the cron.php not running

Question: a particular mail is in error, why?
Answer: There can be multiple reasons, which leads to flagging a mail in error. When browsing the Mailbox Content tab on the Incoming eMail Inboxes the error message should explain you the reason. Attachment too big, mail in an unkown format (encrypted for eg.),…

Question: Change log entries are sometimes empty, also the email replies had valid information, why?
Answer: When using Thunderbird or Gmail, if the user does not enter his reply where the cursor was, but move it first, maybe just one line below, then the text entered is tagged by the mail editor as if it was part of the initial message, embedded in a div tag with a class specific to Gmail or Mozilla

<!-- Example for Thunderbird -->
<div class="moz-cite-prefix">My reply entered after moving the cursor in Thunderbird</div>
 
<!-- Example for Gmail-->
<div class="gmail_quote">My reply after entered moving the cursor in Gmail</div>

To fix this, modify the module configuration and put this parameter

Configuration

   'combodo-email-synchro' => array (
      'html-tags-to-remove' => array(
           'blockquote' => array(), 
           'pre' => array('moz-signature'),
      ),
      // Keep the rest ...

The default behavior when that parameter is not present in the Configuration is the same as putting this:

Default

      'html-tags-to-remove' => array(
           'blockquote' => array(),  // No class specified, remove any blockquote tag
           'div' => array('gmail_quote', 'moz-cite-prefix'),
           'pre' => array('moz-signature'),
      ),

—-

Question: I can't access the mailbox content, I get an error: Reason: cannot read - connection closed?
Answer: If you are trying to access a Mailbox with OAuth2, this message occurs when you have not specified ssl in the imap_options

Configuration

'combodo-email-synchro' => array (
    'imap_options' => array (
         0 => 'imap',
         1 => 'ssl',
    )

In order to recover from the faulty behavior, the version 3.2.1 contains 3 mechanisms to cleanup the useless records in priv_change

1. A periodic cleanup, designed to run in the background during normal day-to-day operations, to cleanup the database step by step,
2. A scheduled cleanup (disabled by default), designed to perform much more massive cleanups during off-hours,
3. A command line script, to be launched by the system administrator, that can count the orphan records and also cleanup a specifed number of records in one shot.

These background processes are executed by the usual background tasks of iTop. They are configured inside the iTop main configuration file, under the section $MyModuleSettings, for the module named combodo-cmdbchange-cleaner.

Example 1: periodic cleanup (only) of up to 1000 records every 2 minutes:

'combodo-cmdbchange-cleaner' => array (
                'scheduled_cleaning_bulk_size' => 0, /* massive cleanup disabled */
                'periodic_cleaning_bulk_size' => 1000,
                'cleaning_periodicity' => 120,
        ),

Example 2: scheduled massive cleanup (only) of up to 100,000 records every Saturday at 3:00 PM:

'combodo-cmdbchange-cleaner' => array (
                'scheduled_cleaning_time' => '15:00',
                'scheduled_cleaning_bulk_size' => 100000,
                'periodic_cleaning_bulk_size' => 0, /* periodic cleanup disabled */
                'scheduled_cleaning_week_days' => 'saturday',
        ),

You can see some information about the cleanup tasks in error.log, for example:

2021-02-10 15:30:33 | Info    | [CMDBChangeScheduledCleaner] Get CMDBChange 10000 ID(s) to remove : executed in 0.160 s | IssueLog
2021-02-10 15:30:33 | Info    | [CMDBChangeScheduledCleaner] Bulk deletion query of 10000 row(s) : executed in 1.603 s | IssueLog
2021-02-10 15:30:33 | Info    | 10000 CMDBChange row(s) deleted. | IssueLog

The script is named change.bulkcleaner.php and if you have the 2.3.1 version of this extension is located in the folder combodo-cmdbchange-cleaner, otherwise you may have received that script alone, in that case we suggest you to put it on the /data directory located under your iTop main directory.

php change.bulkcleaner.php --auth_user=user --auth_pwd=pwd --bulk_size=1000

This produces the following output:

1000 CMDBChange row(s) deleted.

Command line options

For questions related to troubleshooting, check the corresponding section above

Question: We are missing e-mails from users that reply to tickets that have already been closed, what can we do?
Answer: There are different strategies to this issue

1. One solution is to re-open the closed Ticket and reassign it (not nice if the answer just contains a simple “thank you”)
2. Another solution is to activate a notification to the agent on closed tickets with the content of the last log entry, so they can decide what to do with it: ignore or open manually a new ticket. That last action could be facilitate with a User action Configuration entry, which could create a new ticket from the closed one, reusing the last log entry as the description of the new ticket, and copying all other pertinent fields.

Question: Some e-mails update closed tickets. But we want those emails to create new Ticket. How can we do?
Answer: You can force the creation of a new Ticket. For this you must write an extension to overload the method which retrieve the email related Ticket. The idea is to ignore any closed Ticket (see code highlighted below), thus as no closed Ticket will be retrieved, a new Ticket will be created.

MailInboxStandard

protected function GetRelatedTicket(EmailMessage $oEmail)
{  // First check if there is any iTop object mentioned in the headers of the eMail (done by parent class)
   $oTicket = parent::GetRelatedTicket($oEmail);
   if ($oTicket == null) {
      // No associated ticket found by parsing the headers, check
      // if the subject does not match a specific pattern
      $sPattern = $this->FixPattern($this->Get('title_pattern'));
      if(($sPattern != '') && (preg_match($sPattern, $oEmail->sSubject, $aMatches))) {
         $oFilter = DBSearch::FromOQL('SELECT Ticket WHERE ref = :ref AND operational_status != "closed"');
         foreach ($aMatches as $sMatch) {
            if (!empty($sMatch)) {
               $this->Trace("iTop Simple Email Synchro: Retrieving ticket $sMatch (match by subject pattern)...");
               $oSet = new DBObjectSet($oFilter, array(), array('ref' => $sMatch));
               $oTicket = $oSet->Fetch();
               if ($oTicket) {
                  break;
               }
            }
         }
      }
   }                              
   return $oTicket;
}

For Combodo's customers, this can be done easily with the ITSM Designer

Question: Is there a way to disable Mail to Ticket automatically deleting e-mails when closing tickets?
Answer: Not easy. A solution could be to set the Log in read-only when the Ticket is closed, as a result, when a new incoming mail related to that Ticket is processed, it fails to update the ticket, it is flagged in error and kept in the Inbox…

Question: Can I move manually into another folder emails which have been processed?
Answer: When migrating to the version which allows to move processed emails to another mail folder automatically after processing, you may want to do it for older emails. You can do it, but remember that moved emails are no more visible from iTop mailbox content, so only move the processed ones, not the new ones, and maybe not those which were in error.

Question: Can I use mailboxes imposing OAuth 2.0 authentication, such as Gmail or Azure Office 365?
Answer: This requires version 3.6.0 of this extension, as well as an iTop 2.7.7+ or iTop 3.0.2+

For this, you must configure OAuth client first. Then clone your current Mailbox (see tip below for doing this) or create from scratch a OAuth2 Mailbox using that OAuth client

Question: How to simply migrate my standard inbox into an OAuth one?
Answer: Add the objet-copier rule below in your Configuration file, then use the added menu

Configuration

'itop-object-copier' => array (
  'rules' => array (
     'MailInboxStandardToMailInboxOAuth' => array(
        'source_scope' => 'SELECT MailInboxStandard WHERE finalclass = "MailInboxStandard"',
        'allowed_profiles' => 'Administrator',
        'menu_label' => 'Create OAuth 2.0 Mail Inbox...',
        'menu_label/FR FR' => 'Créer une Boite mail OAuth 2.0...',
        'form_label' => 'Create new OAuth 2.0 Mail Inbox from %1$s',
        'form_label/FR FR' => 'Nouvelle Boite mail OAuth 2.0 depuis %1$s',
        'report_label' => 'Created from %1$s',
        'report_label/FR FR' => 'Créée depuis %1$s',
        'dest_class' => 'MailInboxOAuth',
        'preset' => array(
                0 => 'clone(server, mailbox,login,port,behavior, target_class, ticket_default_values, ticket_default_title, title_pattern,unknown_caller_behavior,unknown_caller_rejection_reply,caller_default_values,error_behavior, notify_errors_to, notify_errors_from,trace,email_storage,target_folder,import_additional_contacts,stimuli )',
                1 => 'set(active,no)',
        ),
        'retrofit' => array(),
      ), 

Question: My gmail messages with OAuth2 are not moved to another folder, why?
Answer: There are some configuration on gmail settings to do, for emails to be correctly moved in other folder.

That configuration might solve other problems as well, so you'd better apply this on your gmail account:

First click here: Then on “See all settings”:

Then choose tab “Forwarding and POP/IMAP”: And finally set this configuration:

Question: Parsing messages in my OAuth Mailbox is very slow, what could I do about it?
Answer: We have notice that with laminas, the new library used to poll Mailbox through OAuth2 authentication, the response time is dependent on the size of the parsed folder (Inbox in general). To prevent this situation, we recommend to either delete the email once processed or at least to move them to another folder.

Question: I can't access the mailbox content, I get an error: Reason: cannot read - connection closed?
Answer: If you are trying to access a Mailbox with OAuth2, this message occurs when you have not specified ssl in the imap_options

Configuration

      'combodo-email-synchro' => array (
                // ...
 
                'imap_options' => array (
                     0 => 'imap',
                     1 => 'ssl',
                ),
 
                //...