Ticket Creation from eMails
This extension runs in the background to scan the defined mail inbox(es) and either create or update tickets based on the content of the incoming emails.
Features
-
Connect to any mailbox using either the POP3 or IMAP protocol
-
Interactive configuration of the mail inboxes
-
Processes the incoming eMails in both HTML or plain text format
-
Determines whether a Ticket must be created or updated based either on the custom headers added by iTop in the eMail (in case of replies) or based on a configurable pattern in the title
-
Support both Incidents and User Request tickets
-
Ticket's attachments are automatically turned into attachments of the Ticket (“dangerous” types of attachments can be excluded)
-
Automatic detection of duplicate attachments (like signature images)
-
Keeps the messages in the mailbox until the corresponding Ticket is either closed or deleted
-
Manual retry in case an error occurs when processing an eMail
-
Images embedded inside an HTML mail are displayed inline in iTop as well.
-
Images “too small” (below configurable dimensions) are not imported as attachments (to exclude signatures)
-
Images bigger than configurable dimensions can be resized automatically before uploading into iTop
-
“Inline images” are displayed at a reduced (configurable) size and can be “zoomed-in” by clicking on them.
-
Automatically reject “Autoreply” emails, based on a set of configurable patterns to be tested against the subject of the email
Revision History
Version | Release Date | Comments |
---|---|---|
2.6.12 | 2016-06-07 | Security: only administrators can see the password of mail inboxes. Regression: properly import all attachments (not only the last one if it's not an image). Enhancement: preserve hyperlinks when converting from HTML to plain text. |
2.6.11 | 2016-02-02 | Developers only: fixed a compatibility issue with alternatives to the module itop-standard-email-synchro. The regression has been introduced in 2.6.6 and does not impact the behavior of the component unless you have developped your own alternative to itop-standard-email-synchro. |
2.6.10 | 2015-10-28 | Automatically reject “Autoreply” emails (cf
undesired-subject-patterns below). Support of emails
with no subject. |
2.6.9 | 2015-09-29 | Properly initialize ENUM values (#1102), prevent updating tickets of a different class than the one configured in the mail inbox. |
2.6.8 | 2015-03-09 | Fixed the processing of the
images_maximum_size parameter to prevent errors when
the images are not to be resized. Fixed the pattern for detecting
blockquote tags inside HTML replies. |
2.6.7 | 2015-03-05 | Suppressed a warning when ignoring small images. Change the default value for “body_parts_order”. |
2.6.6 | 2015-01-21 | Support of inline images, filtering of “too small” images and resizing of “big” images. |
2.6.5 | 2014-08-01 | New configuration parameter to workaround a problem with Gmail/IMAP. |
2.6.4 | 2014-07-21 | Enhancement: allow to delete the emails from the server immediately after processing them. French translation of the “TriggerOnMailUpdate” and make the trigger importable. |
2.6.3 | 2014-06-04 | Enhancement: support the creation of Change and Problem tickets. |
2.6.2 | 2014-04-09 | Bug fix in order to support mailboxes containing a backslash in their name (likely to happen using IMAP). |
2.6.1 | 2014-04-08 | Addition of the German localization (though it's not 100% translated). |
2.6.0 | 2014-03-05 | Improved error processing (e.g. keeping the errors in the mailbox, manual retry) and better decoding of the “new part” of the message when it is a reply. Enhanced HTML to text conversion, fixed the processing of “Outlook's forwarded messages” as attachments. |
2.5 | n/a | Support of several mail inboxes. Interactive configuration of the mail inboxes via the iTop user interface. Various improvements for the parsing/decoding of the eMails |
2.2 | 2013-07-22 | “Legacy” version supporting only one mailbox (configured from the iTop configuration file). The documentation for this version is available here: Ticket Creation from eMails (legacy) |
Limitations
-
Each mail inbox configured corresponds to one type of Ticket (i.e. User Requests or Incidents, but not both).
-
Connecting to a POP3 mailbox via the php IMAP extension does not work. You must use the POP3 configuration via PEAR::NetSocket for such a case.
-
There is no validation that the target class of the Mail Inbox is an existing class in iTop. (i.e. don't try to create incidents if the Incident Management module is not installed).
-
The support on inline images works only if the preferred order for email parts is configured as “text/html,text/plain” (i.e. HTML first), which is now the default for all new installations.
Requirements
-
PHP 5.2.1+ with the IMAP extension enabled if you want to connect to an IMAP server, or PEAR::NetSocket (iTop comes with its own copy of PEAR::POP3) if you want to connect to a POP3 server.
-
A connection to a POP3 or IMAP server with a valid mailbox.
-
cron.php must be running to enable the processing of incoming eMails.
-
For resizing big images, PHP GD must be installed.
Installation
-
Expand the content of the zip file in the “extensions” folder of iTop (two folders will be created:
combodo-email-synchro
anditop-standard-email-synchro
). -
Make sure that the web server has enough rights to read all the files in the “extensions” folder.
-
Launch the setup program, and when prompted for the list of extensions to install, check “Ticket Creation from Emails (Standard)” from the list
sudo apt-get install php5-imap sudo php5enmod imap sudo service apache2 restart
Data Mapping
When creating a new ticket from an incoming email message, the application automatically fills the following fields of the ticket:
-
Title = subject of the mail
-
Description = body of the mail
-
Caller (caller_id) = sender of the mail (identified by her/his email address)
-
Organization (org_id) = organization of the caller
-
Origin (if this field exists on the ticket) = 'mail'
Default (i.e. constant) values must be supplied, via the “Ticket Default Values” setting, for any other mandatory field of the Ticket, otherwise the creation of the Ticket will fail.
When updating an existing Ticket, the application add an entry into the public_log field (configurable, see below), with the “new part” of the message. See below for more explanation about how the “new part” is extracted from the message.
Specifying default values
The syntax for specifying default values is the following:
-
One field per line, with the syntax:
-
field_code:default_value
If the field to initialize is a key to another object (for the
fields like org_id, service_id, etc…) you can specify either the
numeric identifier of the target object (e.g.
service_id:153
) or its name (e.g.
org_id:Demo
), provided that the name is unique.
Configuration
The behavior for each mailbox (which messages to process and whether to create or update a Ticket) is managed via the menu “Incoming eMail Inboxes” in the “Admin tools” section:
Click, “Create a new Mail Inbox” to create a new configuration for a mail inbox. This displays the following form:
Mailbox Configuration
The “Mailbox configuration” defines how the application connects to the mail inbox:
Field | Meaning | Sample Value |
---|---|---|
Mail Server | The IP address or fully qualified hostname of the mail server | 10.153.20.142 or pop3.mycompany.com |
Login | The name of the mail account used for connecting to the mailbox | test@mycompany.com |
Password | The password for the above mentioned account | |
Protocol | The protocol to connect to the mail server: either POP3 or IMAP. | POP3 |
Port | The TCP port to connect to the server. The standard values are 110 for POP3 and xxx for IMAP | 110 |
Mailbox (for IMAP) | The IMAP mailbox (folder) to scan for incoming messages. If omitted the default (root) mailbox will be scanned. This option is ignored when using the POP3 protocol. | |
Active | If set to “Yes”, the inbox will be polled. Otherwise no. | Yes |
Debug trace | Use this setting for tracing all the background operations related to this inbox for debugging and troubleshooting purposes. It is not recommended to activate this option for long periods on production since it tends to generate a lot of output which slows down the server | No |
Emails in Error
This section defines the behavior when an incoming email cannot be processed properly. The email can be either kept in the mailbox (and remembered as an “Error” and no longer processed) or deleted immediately from the mailbox. Furthermore the original message can be forwarded to an administrator (as an attachment) along with some explanation about the cause of the error.
Field | Meaning | Sample Value |
---|---|---|
Behavior | Whether or not the emails processed with an error should be kept in the mailbox. If so, the message will be flagged as “Error” and no longer processed, but still available for reading from the mailbox. | Keep the message in the mailbox |
Forward eMails to | The email address to which to forward the email when an error occurs. The forwarded message contains some explanation about the error and the original email as an attachment. If this address is left empty, the incoming emails which cannot be processed will simply be delted from the inbox without further notice. | itopadmin@mycompany.com |
(From) | The IP address to be used as the “sender” of the error notification. For security reasons, many mail servers do not relay messages if the sender address is not a known address. | itopadmin@mycompany.com |
Behavior on Incoming eMails
This section defines the behavior of the application when processing incoming emails.
Field | Meaning | Sample Value |
---|---|---|
Behavior | The behavior when a new message arrives in the
inbox. The possible values are: Create or Update: create a new Ticket or Update an existing one if a matching Ticket is found Create new ticket: each new message creates a new Ticket Update existing tickets: all incoming messages which do not match an existing Ticket are treated as errors. |
Create or Update |
After processing the eMail | The action to be taken after successfully processing an incoming eMail: either keep the eMail on the mail server (until the associated ticket is closed or deleted) or delete the eMail immediately. | Keep the eMail on the mail server |
Ticket Class | The class of Tickets to create or update when receiving an email. Make sure that you select a valid class for your iTop configuration. | User Request |
Ticket default Values | The syntax for “Ticket Default Values” and “New
Person's Default Values” are: - one field to initialize per line - <field_code>:<constant_value> |
service_id:Networking |
Default Title (if subject is empty) | The value to be used as the title of the Ticket, if the subject of the incoming email is empty. If this field is left empty the system will supply a default value (“No Subject”) | Empty subject |
Title pattern | Each notification sent by the application contains a reference to the “source” Ticket in the MessageID field of the email. Email client applications generally store this identifier in the “in-reply-to” or “references” header of the reply email. This is the primary mean of identifying that an email message is related to a ticket. If this header is not present in the incoming message, the application can parse the “subject” field to look for a given match. This pattern determines how to parse the subject. The pattern specified here must follow the PCRE syntax. | /R-([0-9]{6})/ |
Unknown Callers
This section determines the behavior of the application when the sender of an email (From:) does not correspond to a known email address in the application. There are two possibilites:
-
Reject the eMail: the incoming email is treated as an error and thus either forwarded to an administrator or deleted.
-
Create a new person: a new Person will be created based on the email of the sender and constant values defined below.
Field | Meaning | Sample Value |
---|---|---|
Behavior in case of Unknown Caller | What to do when the sender of the incoming email message does not correspond to any Person recorded in the application | Create a new person |
New Person's Default Values | Default values for initializing the new Person.
The application automatically fills the email field
with the email address of the sender of the message. All other
mandatory fields must be initialized with constant values provided
here, otherwise the creation of the new Person will fail. |
first_name:Unknown name:Caller org_id:Demo |
Other configuration parameters
In addition to the configuration performed by creating a Mail Inbox object using the user interface of the application, a few parameters are available in the configuration file to fine tune the behavior of the application.
body_parts_order
is
text/html,text/plain
. Indeed in order to properly
import the images embedded inside an HTML email, the HTML version of the email must
be processed instead of the plain text version.'combodo-email-synchro' => array ( 'debug' => false, 'periodicity' => 30, 'body_parts_order' => 'text/html,text/plain', 'pop3_auth_option' => 'USER', 'imap_options' => array ( 0 => 'imap', ), 'exclude_attachment_types' => array ( 0 => 'application/exe', ), 'maximum_email_size' => '10M', 'introductory-patterns' => array ( 0 => '/^le .+ a écrit :$/i', 1 => '/^on .+ wrote:$/i', 2 => '|^[0-9]{4}/[0-9]{1,2}/[0-9]{1,2} .+:$|', ), 'multiline-delimiter-patterns' => array ( 0 => '/\\RFrom: .+\\RSent: .+\\R/m', 1 => '/\\R_+\\R/m', 2 => '/\\RDe : .+\\R\\R?Envoyé : /m', 3 => '/\\RDe : .+\\RDate d\'envoi : .+\\R/m', 4 => '/\\R-----Message d\'origine-----\\R/m', 5 => '/\\RExpéditeur: .+\\RDate:/m', 6 => '/\\RDe : .+\\RDate : /m', 7 => '/\\TO:.+\\RCC:/m', ), 'big_files_dir' => '', 'use_message_id_as_uid' => false, // Don't change this unless you know what you are doing! 'images_minimum_size' => '100x20', 'images_maximum_size' => '', 'undesired-subject-patterns' => array ( 0 => '/^Out Of Office/i', 1 => '/^Automatic answer$/i', 2 => '/^Réponse automatique:/', ), ),
Parameter | Meaning | Default Value |
---|---|---|
debug | Set to true to turn on debug output | false |
periodicity | Interval (in seconds) at which to check for incoming messages | 30 |
body_parts_order | Comma separated, ordered, list of MIME types,
determining the preferred part of the message to retrieve for
populating the description or public_log
of the Ticket. Note: new since 2.6.6 In order to
import as inline images the images embedded in HTML, the HTML part of the email must be
processed in priority over the text part. Therefore the recommended
configuration is text/html,text/plain . |
text/html,text/plain |
pop3_auth_options | POP3 authentication options. Possible values are: 'CRAM-MD5', 'APOP' , 'PLAIN' , 'LOGIN', 'USER' | USER |
imap_options | Additional IMAP options. Possible values are
listed here: IMAP flags. Warning: Do NOT use
the pop3 flag to connect to a POP3 mailbox using the
PHP IMAP extension. Due to a limitation of the IMAP extension this
will not work! If you want to connect to a POP3 server, use the
POP3 protocol instead. |
imap |
exclude_attachment_types | Array of MIME types to exclude when retrieving attachments | array('application/attachment') |
maximum_email_size | If an incoming email is bigger than the specified
size, the message will saved to the big_files_dir if
it is configured and deleted from the inbox. A notification message
will be sent to the administrator as for other “errors” happening
when processing the mail inbox. The size can be specified using the
'short' notation: 100K, 3M, 2G… If set to zero, no limit will be
enforced… with the risk of a PHP crash if there is not enough
memory to decode an incoming email. |
10M |
introductory-patterns | When computing the “new part” of a message, lines matching this pattern and preceding what looks like an “old part” of the message, are removed. Adapt this list to your localization… and favorite email client dialect. The pattern specified here must follow the PCRE syntax. |
array ( 0 => '/^le .+ a écrit :$/i', 1 => '/^on .+ wrote:$/i', 2 => '|^[0-9]{4}/[0-9]{1,2}/[0-9]{1,2} .+:$|', ) |
multiline-delimiter-patterns | Multi-line regular expression patterns used for computing the “new part” of a message. each of theses patterns determine the beginning of the “old part” of a message, when a message is a “Reply” to another message. Everything that matches this pattern (and all the text that follows this match) will be removed for the “new part”. All patterns are tested successively. The pattern that provides a match closer to the beginning of the text will be used. Adapt this list to your localization and to the dialect of your favorite email clients… The pattern specified here must follow the PCRE syntax. |
array ( 0 => '/\\RFrom: .+\\RSent: .+\\R/m', 1 => '/\\R_+\\R/m', 2 => '/\\RDe : .+\\R\\R?Envoyé : /m', 3 => '/\\RDe : .+\\RDate d\'envoi : .+\\R/m', 4 => '/\\R-----Message d\'origine-----\\R/m', 5 => '/\\RExpéditeur: .+\\RDate:/m', 6 => '/\\RDe : .+\\RDate : /m', 7 => '/\\TO:.+\\RCC:/m', ) |
big_files_dir | The path to a directory where to store emails
bigger than maximum_email_size . If this directory is
not configured, the emails are simply deleted before notifying the
administrator. |
|
use_message_id_as_uid | Boolean. For IMAP connections only. Whether or not to use the identifier from the message (MessageID) instead of the Mailbox' unique identifier (UID) to uniquely identify the already processed messages. This can be useful to workaround problems when the UID of the messages on the server changes between sessions (like with Gmail). If you toggle this value, make sure that you first empty the mailbox (and stop the cron job), since all messages present in the mailbox when the setting is changed will be considered as new and processed again. | false |
images_minimum_size | new in 2.6.6 Minimum dimensions
for importing images. Images smaller than the given dimensions will
be ignored and not imported as attachments. The dimensions are
expressed as a string widthx height
(where width and height are integer numbers, in
pixels). |
100×20 |
images_maximum_size | new in 2.6.6 Images bigger than
these dimensions (for example 1000×1000 ) will be
resized to fit in the given dimensions. The dimensions are
expressed as a string widthx height
(where width and height are integer numbers, in
pixels). Note this feature is available only if
PHP GD is installed. If no dimensions are given, the images are
never resized. |
|
undesired-subject-patterns | An array of regular expression patterns (as PHP text strings) that will be used to test the subject of the incoming email. If any of these pattern matches, the email will be considered as “undesirable” and rejected (the same processing as for any other error case will then be applied). The patterns specified here must follow the PCRE syntax. | array() |
'itop-standard-email-synchro' => array ( 'ticket_log' => array ( 'UserRequest' => 'public_log', 'Incident' => 'public_log', ), ),
Parameter | Meaning | Default Value |
---|---|---|
ticket_log | An associative array (hash) defining, for each class of Ticket, the code of the attribute to set when updating a ticket from an incoming message. You don't need to change this value unless you modified the Data Model. |
array ( 'ticket_log' => array ( 'UserRequest' => 'public_log', 'Incident' => 'public_log', ), ) |
Checking the connection
Once the Mail Inbox object has been created, you can use the tab “Mailbox Content” in the details of the object to check that the application can properly connect to the mail server and retrieve messages from it.
This view also allows to perform two different kind of actions on a set of messages:
-
“Reset Status”: for messages which are either flagged as “Error” or “Already processed”, this status will be reset and the email will considered again as “New” and thus candidate for processing the next time cron.php runs.
-
“Delete eMail”: deletes the message(s) from the mailbox. No confirmation will be asked !!
Debugging
Since the processing of the incoming emails occurs in the background, it is not always easy to understand what happens when a ticket is not processed as expected. In order to trace the execution of this background task, several levels of tracing are available:
-
by setting the “Debug trace” field to “Yes”, more trace will be generated when processing the mailbox. This trace goes to the output of the cron job, and is also captured (limited to 256 KB) in the database. This captured trace is visible in the “Debug Trace” tab of the Mail Inbox object.
-
the configuration setting
debug
for the modulecombodo-email-synchro
can be set to true in the configuration file. This activates even more traces for all the configured mail inboxes. This additional trace appears only in the output of the cron job. -
the cron job can be passed the optional parameter
--verbose=1
to activate some debug trace for all the background tasks. This additional trace also goes into the output of the cron job.
If the cron job is not already running in the background it is convenient to run it manually from the command line, to see what's happening:
php cron.php --auth_user=<user> --auth_pwd=<pwd> --verbose=1