Data collector for vSphere

This stand-alone application collects information about a whole data center from a vSphere server and synchronizes this information in iTop.

The Data collector for vSphere is a stand-alone PHP software that extracts from vSphere informations related to a set of CIs and synchronizes them with iTop's CMDB.

• Collection of CI informations is done through a VMware / vSphere PHP library,
• Synchronization follows iTop's built-in Data Synchronization mechanism.

The collector currently focuses on the following classes of objects, for which collection can be activated or deactivated on an individual basis:

• Brand
• Model
• OS Family
• OS Version
• Farms
• Server
• Hypervisors
• Virtual Machine
• Logical Interface - Requires TeemIP / IPAM for iTop extension
• IPv4 Address - Requires TeemIP / IPAM for iTop extension
• IPv6 Address - Requires TeemIP / IPAM for iTop extension
• Datastore - Requires iTop extension Data Model for vSphere

Starting with version 1.0.12 the collection mechanism is extensible.

The current version of the collector is designed to collect information from a single vSphere server. To collect and reconcile information from several vSphere servers into one iTop, check the section Collecting from several vSphere servers below.

Usage of the collector requires you to comply with a few points:

• PHP Version 7.4.0 minimum
• An access to the vSphere web services API
• An access to the iTop web services
• The vSphere objects synchronized by the collector must exist in iTop as CMDB CIs.
  • This is the default case for most collected objects if iTop has been installed with the Virtualization Management Module,
  • Logical interfaces, IPv4 and IPv6 Addresses do need TeemIP / IPAM for iTop,
  • Datastores require iTop extension Data Model for vSphere
• From a system standpoint, you'll need to comply with the requirements expressed in the Data collector Base documentation.

Simply expand the content of the zip archive into a folder on the machine where the collector will be run.

The configuration of the application is built by concatenating 4 files:

1. conf/params.distrib.xml that is provided by the collector framework, Data collector Base. It should not be modified.
2. collectors/params.distrib.xml that holds the entries that are specific to the Data collector for Azure. It should not be modified.
3. collectors/extensions/params.distrib.xml that holds entries that have been created and added by the customer to answer its specific needs. This file is optional.
4. conf/params.local.xml where the collector can be adapted to the specific customer needs.

The collectors/params.distrib.xml configuration file holds parameters that must (for some) or can (for others) be changed when configuring the collector, which is done through the conf/params.local.xml file.

<?xml version="1.0" encoding="UTF-8"?><!-- Default values for parameters. Do NOT alter this file, use params.local.xml instead -->
<parameters>
        <!-- vSphere Parameters -->
        ...
           List of parameters to connect to the remote vSphere application
        ...
 
        <!-- Default values -->
        ...
           Default paramaters that drive the collect
        ...
 
        <!-- Class collection sequence -->
        ...
           List of classes to collect with the rank in the collection process
        ...
 
        <!-- Synchronization parameters -->
        ...
           Placeholders for the synchro data sources
        ...
 
        <!-- IPs and logical interfaces collection -->
        ...
           Parameters that dedicated to the collect of TeemIP objects
        ...
 
        <!-- Mapping tables -->
        ...
           Set of mapping tables
        ...
 
</parameters>

The collectors/extensions/params.distrib.xml configuration file holds parameters to collect additional classes required by a customer.

<?xml version="1.0" encoding="UTF-8"?><!-- Default values for parameters. Do NOT alter this file, use params.local.xml instead -->
<parameters>
        <!-- Default values -->
        ...
           List of default values for the additional classes to collect
        ...
 
        <!-- Class collection sequence -->
        ...
           List of additional classes to collect with the rank in the collection process
        ...
 
</parameters>

This set of parameters is required to connect to the iTop application or the vSphere environment. Some of them must or may be adapted to meet customers' own environment.

  <vsphere_connection_options>
        <ssl>
                <verify_peer>0</verify_peer>
                <verify_peer_name>0</verify_peer_name>
                <allow_self_signed>1</allow_self_signed>
        </ssl>
  </vsphere_connection_options>

The section defines a set of default general parameters for the collects.

  <!-- Default values -->
  <default_org_id>Demo</default_org_id>
  <hypervisor type="hash">  
    <!-- Define what attribute to use as cpu attribute within iTop's datamodel.
     numCpuPackages was used before -->
    <cpu_attribute>numCpuCores</cpu_attribute>
  </hypervisor>
  <virtual_machine type="hash">
    <!-- By default, a VM's virtual host points to the farm that the VM's hypervisor belongs to, if any.
        This option forces the VM's virtual host to point to the VM's hypervisor
     Allowed values: hypervisor, farm (default) -->
    <virtual_host>farm</virtual_host>
  </virtual_machine>

This section defines the list of classes that will be collected and in which order. It enables as well the possibility to deactivate the collection of a class.

  <collectors_launch_sequence type="array">
    <!-- Brands -->
    <collector>
      <name>vSphereBrandCollector</name>
      <enable>yes</enable>
      <rank>1</rank>
    </collector>
    <!-- Models -->
    <collector>
      <name>vSphereModelCollector</name>
      <enable>yes</enable>
      <rank>2</rank>
    </collector>
    <!-- OS Families -->
    <collector>
      <name>vSphereOSFamilyCollector</name>
      <enable>yes</enable>
      <rank>3</rank>
    </collector>
    <!-- OS Versions -->
    <collector>
      <name>vSphereOSVersionCollector</name>
      <enable>yes</enable>
      <rank>4</rank>
    </collector>
    <!-- Farms -->
    <collector>
      <name>vSphereFarmCollector</name>
      <enable>yes</enable>
      <rank>10</rank>
    </collector>
    ...
  </collectors_launch_sequence>

The collector automatically detects if TeemIp is present on the remote iTop application (as a module or even as a stand-alone application). Should that be the case, then the following parameters may trigger IP addresses and logical interfaces collection.

<?xml version="1.0" encoding="UTF-8"?>
  <parameters>
    ...
    <teemip_discovery type="hash">
      <collect_ips>yes</collect_ips>
      <default_ip_status>allocated</default_ip_status>
      <manage_ipv6>no</manage_ipv6>
      <manage_logical_interfaces>yes</manage_logical_interfaces>
      </teemip_discovery>
    ...
  </parameters>

These are dedicated to the synchronization data sources that the collector creates. Some of them must or may be adapted to meet customers' own environment.

This section groups the mapping tables used by the collector.

Should you require further details on how mapping tables should be built and use, please refer to the Data mapping chapter of the Data collector Base documentation.

Starting with version 1.0.12 of the application, Server and Hypervisor collectors are configurable by adding extra definitions in the XML parameter file.

This configuration contains two parts:

1. The first part (under the <json> tag) allows to alter the definition of the Synchro Data Source (normally fixed by the .json file associated with the collector class). This is useful to synchronise a new field which was not synchronized by default, or to change a reconciliation rule on a field, etc… Any field under the attribute_list part of the JSON file can be modified by specifying a new value in the configuration.
2. The second part (the <source> tag) defines the way to collect the data for the attribute. This tag must contain a (partial) PHP expression relative to the Hypervisor object (which is actually a HostSystem in the vSphere model)

Both parts of the configuration can be used independently (one can alter only the JSON or only the collection, or both).

The example below configures the collection of the “Serial Number” on Servers (from the otherIdentifyingInfo[EnclosureSerialNumberTag]) and also uses the collected serial number as the reconciliation key between Hypervisors and physical Servers (the field server_id).

config.local.xml

<?xml version="1.0" encoding="UTF-8"?>
<parameters>
 
  ...
 
  <custom_synchro>
    <vSphereHypervisorCollector>
      <fields>
        <server_id>
          <source>hardware->systemInfo->otherIdentifyingInfo[EnclosureSerialNumberTag]</source>
          <json>
            <reconciliation_attcode>serialnumber</reconciliation_attcode>
          </json>
        </server_id>
      </fields>
    </vSphereHypervisorCollector>
    <vSphereServerCollector>
      <fields>
        <serialnumber>
          <source>hardware->systemInfo->otherIdentifyingInfo[EnclosureSerialNumberTag]</source>
        </serialnumber>
      </fields>
    </vSphereServerCollector>
  </custom_synchro>
 
  ...
 
</parameters>

The launch of the vSphere collector will be driven by the command and parameters defined in the usage section of iTop Data collector base. Once launched, first action of the collector will be to define its collection plan, based on the list of classes that have been enabled in the configuration file. Then:

• Configuration files will be consolidated,
• Synchronisation data sources will be created or updated if required,
• Collection of vSphere classes will be made and extracted data will be stored under the local “data” directory, in CSV format,
• vSphere objects will be uploaded into iTop,
• Synchronisation of the collected data with the existing iTop CIs will be made.

If you have troubles connecting to the vSphere server, try the following troubleshooting steps:

1. Check that the connection from the system running PHP to the vSphere server is currently possible. Use a command line tool like wget to connect to the vSphere server. For example if your vsphere_uri is 192.168.10.12:443, try:

   wget --no-check-certificate -O - https://192.168.10.12:443

   If the connection does not succeed, there may be a firewall blocking your requests, or that vSphere is configured to operate on a different port (for example 9443)… ask your IT department about it.

2. Once the connection seems to go fine, configure the proper vsphere_uri value in conf/params.local.xml and, from the command line (in the collectors subdirectory) launch:

   php check_soap.php

   The expected output (with vsphere_uri = 192.168.10.12:443) is:

   Connecting to https://192.168.10.12:443/sdk
   
   Ok, the response looks like a valid SOAP response.
   
   --------------------- DEBUG ----------------
   The request returned:
   <?xml version="1.0" encoding="UTF-8"?>
   <soapenv:Envelope xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
    xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
   <soapenv:Body>
   <RetrieveServiceContentResponse xmlns="urn:vim25"><returnval><rootFolder type="Folder">group-d1</rootFolder><propertyCollector type="PropertyCollector">propertyCollector</propertyCollector><viewManager type="ViewManager">ViewManager</viewManager><about><name>VMware vCenter Server</name><fullName>VMware vCenter Server 4.1.0 build-345043</fullName><vendor>VMware, Inc.</vendor><version>4.1.0</version><build>345043</build><localeVersion>INTL</localeVersion><localeBuild>0</localeBuild><osType>win32-x86</osType><productLineId>vpx</productLineId><apiType>VirtualCenter</apiType><apiVersion>4.1</apiVersion></about><setting type="OptionManager">VpxSettings</setting><userDirectory type="UserDirectory">UserDirectory</userDirectory><sessionManager type="SessionManager">SessionManager</sessionManager><authorizationManager type="AuthorizationManager">AuthorizationManager</authorizationManager><perfManager type="PerformanceManager">PerfMgr</perfManager><scheduledTaskManager type="ScheduledTaskManager">ScheduledTaskManager</scheduledTaskManager><alarmManager type="AlarmManager">AlarmManager</alarmManager><eventManager type="EventManager">EventManager</eventManager><taskManager type="TaskManager">TaskManager</taskManager><extensionManager type="ExtensionManager">ExtensionManager</extensionManager><customizationSpecManager type="CustomizationSpecManager">CustomizationSpecManager</customizationSpecManager><customFieldsManager type="CustomFieldsManager">CustomFieldsManager</customFieldsManager><diagnosticManager type="DiagnosticManager">DiagMgr</diagnosticManager><licenseManager type="LicenseManager">LicenseManager</licenseManager><searchIndex type="SearchIndex">SearchIndex</searchIndex><fileManager type="FileManager">FileManager</fileManager><virtualDiskManager type="VirtualDiskManager">VirtualDiskManager</virtualDiskManager></returnval></RetrieveServiceContentResponse>
   </soapenv:Body>
   </soapenv:Envelope>
   ---------------------------------------------

   If the result is not the expected one, check that your vSphere server is properly configured to run as HTTPS (which should be the default).

In the vSphere web services SDK, Hypervisors and physical Servers are represented by the same object HostSystem.

The information from the vSphere HostSystem object is imported in iTop into the Server object using the following mapping:

The information from the vSphere HostSystem object is imported in iTop into the Hypervisor object using the following mapping:

In the vSphere web services SDK, a Farm is represented by the object ClusterComputeResource.

The information from the vSphere ClusterComputeResource object is imported in iTop into the Farm object using the following mapping:

The list of hypervisors belonging to a farm is collected via the property: ClusterComputeResource → host → name.

In the vSphere web services SDK, a Virtual Machine is represented by the object VirtualMachine. The information from the vSphere VirtualMachine object is imported in iTop into the VirtualMachine object using the following mapping:

If enabled (TeemIp / IPAM for iTop required), the information from the vSphere IPv4 and IPv6 addresses is imported in iTop into the IPv4Address or IPv6Address objects using the following mapping:

Regular expressions are used to discriminate between the two IP formats.

If enabled (TeemIp / IPAM for iTop required), the collector may register the logical interfaces attached to virtual machines. These are imported in iTop into the LogicalInterface objects using the following mapping:

In the vSphere web services SDK, a Datastore is represented by the object Datastore. If enabled (Data model for vSphere extension required), the information from the vSphere Datastore object is imported in iTop into the Datastore object using the following mapping:

Question: how can I add vSphere classes to the collection plan
Answer: Other classes than the ones listed by default in the collector can be added to it. For that purpose, we highly recommend to use the extensibility mechanism that is available since version 2 of the collector (version 1.3.0 of iTop Data Collector Base).

• Make sure the class (MyNewClass) has been modelized in iTop (Through the Data model for vSphere extension for instance).
• Within the <your_collector>/collectors/extensions/ directory:
  • Create a params.distrib.xml file
    • Add default parameter for MyNewClass, if needed,
    • Add a extensions_collectors_launch_sequence similarly to what is done in the Class Collection Sequence section,
  • Create the json/MyNewClassvSphereCollector.json file that describes the synchro data source of the new class,
  • Create the src/MyNewClassvSphereCollector.class.inc.php file that contains the collector specificities for the class,
• And that's it 🙂

You may refer to what already exists for other classes as examples.