Objective 5.4 – Configure, manage, and analyze vSphere and SSO log files

Generate vCenter Server and ESXi log bundles

Using the vSphere client, either connected to vCenter or directly to an ESXi host you can Export System Logs, from the option on the File menu:

support01

 

You can then choose what diagnostic data you wish to export:

support02

 

The next screen will allow you to choose a download location, then the logs will be gathered and downloaded.

You can also generate log bundles using PowerCLI by running the following command once you are connected to either vCenter or a host:

Get-Log -Bundle -DestinationPath <path>

support10

Once its complete you will have the logs collected into a single .tgz file:

support03

Another way of generating host log bundles is by running the vm-support script. You can do this via a SSH or console connection to a host, by running vm-support, these can then be collected using winscp or something similar:

support04

There are various options/switches you can specify when running vm-support, as shown below:

/var/log/vmware # vm-support -h

support05

Use esxcli system syslog to configure centralized logging on ESXi hosts

After establishing a connection to the host, the first thing to do is to check the current configuration:

support06

To set the remote host to log to you can run:

esxcli system syslog config set --loghost syslog.vcrooky.com

support11

This host is logging to the syslog collector installed on the a seperate Windows server (syslog.vcrooky.com).

support12

 

You can get more granular details on the different logs by running:

support13

After making changes, it is recommended that you reload the syslog daemon:

# esxcli system syslog reload

If you have set up your hosts to log to a remote syslog collector but the logs aren’t showing up, then you should check your hosts firewall configuration to ensure that the syslog ports are open:

support09 You could also set this using esxcli by running:

esxcli network firewall ruleset set -r syslog -e true

You can also configure syslog in the vSphere client by accessing the host’s advanced settings, and selecting ‘syslog’:

support08
Test centralized logging configuration

To test your syslog configuration you can ‘mark’ all logs with a custom message by running:

~ # esxcli system syslog mark –message “SyslogTest”

support14

Analyze log entries to obtain configuration information

cd into /var/log, and have a look at the logfiles available. Use the commands: tail, grep, more or vi to browse the log files.

  • /var/log/syslog.log
  • /var/log/vmkernel.log
  • /var/log/vmkwarning.log, contains a summary of warnings and alert log messages from the vmkernel.log

# grep rescan vmkernel.log

support15

Analyze log entries to identify and resolve issues

While investigating an issue, it is a good idea to analyze log files, like the hostd.log or vmkernel.log for specific messages. Those messages can help you finding a VMware KB that can solve your issue or contacting a colleague or VMware Support.

Install and configure VMware syslog Collector and ESXi Dump Collector

  • The Syslog Collector can be installed on the vCenter Server or on a separate server that has a network connection to the vCenter Server.
  • The Syslog Collector does not support IPv6.
  • The product is on the same media as the vCenter Server
  • The installation is pretty straightforward. During the installation you can adjust parameters, like;
    • Location where to install
    • Location for the Syslog Repository
    • Max. size of the repository
    • Max.number of log rotations to keep
    • Protocols and Ports to be used and whether secure connections (SSL) should be used

You can configure ESXi to dump the vmkernel memory to a network server, rather than to a disk, when the system has encountered a critical failure. Install vSphere ESXi Dump Collector to collect such memory dumps over the network.

In the vCenter Appliance, the ESXi Dump Collector is enabled by default. This section applies to Windows based environments.

  • The ESXi Dump Collector can be installed on the vCenter Server or on a separate server that has a network connection to the vCenter Server.
  • The ESXi Dump Collector does not support IPv6.
  • The product is on the same media as the vCenter Server
  • The installation is pretty straightforward. During the installation you can adjust parameters, like;
    • Location where to install
    • Server Port to be used, default is 6500.

support16

One remarkable note from the documentation:

If you configure an ESXi system that is running inside a virtual machine that is using a vSphere standard switch, you must choose a VMkernel port that is in promiscuous mode. ESXi Dump Collector is not supported on vSphere distributed switches.

In this example, vmk0 is VMkernel NIC for management; 192.168.0.205 is the server with ESXi Dump Collector installed.

support17

So how do we test a PSOD? vsish -e set /reliability/crashMe/Panic (Warning: will PSOD your ESXi host!)

support18