SSH authentication logs:

ESXi host agent logs:

ESXi Shell logs:

vCenter Server agent logs:

Per virtual machines logs:

VMware ESXi access

ESXi host can be accessed and managed using a number of methods.

Access method Description Port
ESXi Host Client Web management interface accessible over HTTPS. Access through the web interface are often leveraged to enable SSH. TCP 443
Secure Shell (SSH) SSH access to execute commands locally on the ESXi host through the ESXi Shell. SSH is commonly leveraged by threat actors to perform operations on the ESXi host, including power-off and encryption of virtual machines’s virtual disk vmdk for ransomware operators. 22
Direct Console User Interface (DCUI) Text-based console accessible from the physical console attached to the ESXi host. -
vSphere Web Services API API exposed as a Web service to manage vCenter instances and ESXi hosts, including life-cycle operations on virtual machines. 443

Microsoft Active Directory Domain Services integration

ESXi hosts can be joined to an Active Directory domain. If an ESXi host is joined to an AD domain, members of the AD group defined in the Config.HostAgent.plugins.hostsvc.esxAdminsGroup parameter will have full administrative access to the host. By default, the esxAdminsGroup parameter is set to ESX Admins.

The esxAdminsGroup parameter can be consulted and set in the ESXi Host Client web management interface: “Managed” -> “Advanced Settings” -> “Config.HostAgent.plugins.hostsvc.esxAdminsGroup”. The value is stored in the /etc/vmware/configstore/current-store-1 configuration file. Additionally, for offline analysis, the esxAdminsGroup parameter value is retrieved in a support bundle collection .

VMware ESXi logs


VMware ESXi (5.0 and higher) run a Syslog service (vmsyslogd), that logs messages from the VMkernel and other system components.

The log location is defined through the logdir variable in the vmsyslogd service’s configuration file /etc/vmsyslog.conf. By default, the logs are placed on a local scratch volume, or a ramdisk, mapped to /scratch/log/ / /var/run/log/. The folders /scratch/log/ and /var/run/log/ are symlinks pointing to the same /vmfs/volumes/<UID>/log/ directory. Some minor log files, and symlinks to log files under /vmfs/volumes/<UID>/log/, are also placed in the /var/log/ folder.

The <LOG_DIR> below can be mapped to /scratch/log/, /var/run/log/, /vmfs/volumes/<UID>/log/, or a custom directory depending on how the logs are configured and accessed / collected.

Component Location Details
Authentication <LOG_DIR>/auth.log Events related to authentication to the local ESXi host.

Includes events from the sshd daemon for remote access.
ESXi host agent <LOG_DIR>/hostd.log Events from the vmware-hostd host / management daemon, that is responsible for most of the operations on the ESXi host and its associated virtual machines.

Include events:
- On authentications, for SSH and other access types that are not recorded in auth.log, such as access through the web management interface ESXi Host Client, PowerCLI, programming language SDK, etc.
- On virtual machine life-cycle operations, such as VM creation, power on / off, deletion, snapshot operations.
- On PowerCLI guest operations, such as files transfer and command executions notably.
- etc.
ESXi Shell <LOG_DIR>/shell.log Events for shell related activity.

Include events for every commands entered in the ESXi Shell, shell session lifecycle, enabling / disabling of SSH access, etc.
Virtual machines /vmfs/volumes/<DATASTORE_GUID>/<VM>/vmware.log Per virtual machine events on the VM life-cycle operations.

Include events for VM configuration changes, snapshot operations, Vmware WebMKS console access, guest operations (files transfer and command executions notably) through PowerCLI, etc.
vCenter Server agent <LOG_DIR>/vpxa.log Events related to the ESXi agent that communicates with vCenter Servers (if the ESXi host is managed by vCenter).
Install and updates <LOG_DIR>/esxupdate.log Install and updates related events.

Additionally, for environments with vCenter, VI events can also be a great source of information (on every vSphere API calls made through vCenter).

Authentication to the ESXi host

Source: auth.log

2023-12-05T17:35:43.281Z In(38) sshd[1053654]: Connection from port 58470
2023-12-05T17:35:43.320Z In(38) sshd[1053654]: Accepted keyboard-interactive/pam for root from port 58470 ssh2
2023-12-05T17:59:31.981Z In(38) sshd[1053654]: Session closed for 'root' on /dev/char/pty/t0
Source: hostd.log

2023-12-05T14:31:48.557Z In(166) Hostd[1050748]: [Originator@6876 sub=Vimsvc.HaSessionManager opID=esxui-8c02-9b39 sid=52f0e556] Accepted password for user root from - session=52f0e556-5be4-83d4-8d71-8f9d8a43791c
2023-12-05T14:31:48.557Z In(166) Hostd[1050748]: [Originator@6876 sub=Vimsvc opID=esxui-8c02-9b39 sid=52f0e556] [Auth]: User root
2023-12-05T14:31:48.558Z In(166) Hostd[1050748]: [Originator@6876 sub=Vimsvc.ha-eventmgr opID=esxui-8c02-9b39 sid=52f0e556] Event 99 : User [email protected] logged in as Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ Safari/537.36 Edg/
2024-06-03T19:58:26.435Z In(166) Hostd[1050278]: [Originator@6876 sub=Vimsvc.ha-eventmgr opID=c4c96306 sid=52018116] Event 220 : User [email protected] logged in as PowerCLI/

Operations on the ESXi host itself

Source: auth.log

2023-12-05T17:35:29.040Z No(13) SSH[1053644]: SSH login enabled
2023-12-05T17:35:29.106Z No(13) SSH[1053649]: SSH sandbox is not enabled, SSH will run in superdom
Source: hostd.log

2023-12-05T17:35:29.045Z In(166) Hostd[1050749]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 129 : SSH access has been enabled.
2023-12-05T17:35:29.163Z In(166) Hostd[1050753]: [Originator@6876 sub=Hostsvc.ServiceSystem opID=esxui-42e-a241 sid=52f0e556 user=root] TSM-SSH running status is true
2023-12-05T17:35:29.163Z In(166) Hostd[1050753]: [Originator@6876 sub=Vimsvc.ha-eventmgr opID=esxui-42e-a241 sid=52f0e556 user=root] Event 130 : SSH for the host has been enabled
Source: shell.log

2023-12-05T17:36:17.835Z In(14) shell[1053659]: [root]: vim-cmd vmsvc/getallvms
2023-12-05T17:37:35.560Z In(14) shell[1053659]: [root]: esxcli system version get
2023-12-05T17:37:45.836Z In(14) shell[1053659]: [root]: esxcli system account list
2023-12-05T17:38:21.798Z In(14) shell[1053659]: [root]: esxcli vm process list
2023-12-05T17:42:32.948Z In(14) shell[1053659]: [root]: vim-cmd vmsvc/snapshot.create $vm_id
Source: hostd.log

2024-06-02T21:19:01.386Z In(166) Hostd[1050738]: [Originator@6876 sub=Hostsvc.AppConfigOptionsProvider(Config.HostAgent.) opID=esxui-a003-5e51 sid=5295925c user=root] Set called with key 'Config.HostAgent.plugins.hostsvc.esxAdminsGroup' value '"ESX Admins new"'

Operations on Guest virtual machines

Source: shell.log (if conducted through ESXi shell)

2023-12-05T17:42:32.948Z In(14) shell[1053659]: [root]: vim-cmd vmsvc/snapshot.create $vm_id
Source: /vmfs/volumes/<DATASTORE_GUID>/<VM>/vmware.log (of the VM that has been snapshot)

2023-12-05T17:42:40.666Z In(05) vmx vim-cmd-09-a45d SnapshotVMX_TakeSnapshot start: 'snapshot_exfill', deviceState=1, lazy=1, quiesced=0, forceNative=0, tryNative=1, saveAllocMaps=0
2023-12-05T17:42:40.680Z In(05) vcpu-0 vim-cmd-09-a45d MainMem: Writing full memory image, '/vmfs/volumes/643a85c2-3920a51f-8175-48210b361288/Base-WindowsServer2019/Base-WindowsServer2019-Snapshot2.vmem'
2023-12-05T17:42:40.741Z In(05) vcpu-0 - SnapshotVMXTakeSnapshotWork: Initiated lazy snapshot 'snapshot_exfill': 2
2023-12-05T17:42:41.585Z In(05) vmx - SnapshotVMXTakeSnapshotComplete: Done with snapshot 'snapshot_exfill': 2
Source: /vmfs/volumes/<DATASTORE_GUID>/<VM>/vmware.log (of the VM that has been snapshot)

2023-12-05T17:59:05.961Z In(05) vmx - MKSVMX: Vigor requested a screenshot
2023-12-05T17:59:05.961Z In(05) svga - MKSScreenShotMgr: Taking a screenshot
Source: /vmfs/volumes/<DATASTORE_GUID>/<VM>/vmware.log (of the VM that has been snapshot)

2023-12-05T18:02:22.340Z In(05) vmx - VigorTransportProcessClientPayload: opID=e0ada929 seq=8511: Receiving MKS.IssueTicket request.
2023-12-05T18:02:22.341Z In(05) vmx e0ada929 Issuing new webmks ticket 2d0378... (120 seconds)
2023-12-05T18:02:23.807Z In(05) mks - Accepting connection for webmks ticket 2d0378...
2023-12-05T18:02:23.807Z In(05) mks - MKSRemoteMgr: AddRemoteConsole (numConsoles=0)
2023-12-05T18:02:23.807Z In(05) mks - SOCKET 7 (119) Creating VNC remote connection.
2023-12-05T18:02:23.807Z In(05) svga - MVNCEncode: Number of screens changed from 0 to 1
Source: hostd.log

2024-06-03T22:46:09.892Z In(166) Hostd[1050233]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 229 : Guest operation Initiate File Transfer To Guest performed on Virtual machine <VIRTUAL_MACHINE_NAME>.
2024-06-03T22:46:09.969Z In(166) Hostd[1050237]: [Originator@6876 sub=Guestsvc.GuestFileHandler] Guest File Path is <DESTINATION_FULL_PATH>
Source: /vmfs/volumes/<DATASTORE_GUID>/<VM>/vmware.log (of the VM to which the file was transferred)

2024-06-03T22:45:46.667Z In(05) vmx - VigorTransportProcessClientPayload: opID=c4c964e7 seq=845: Receiving GuestOps.InitiateFileTransferToGuest request.
2024-06-03T22:45:46.679Z In(05) vcpu-1 - VigorTransport_ServerSendResponse opID=c4c964e7 seq=845: Completed GuestOps request.
Source: hostd.log

2024-06-04T21:53:35.264Z In(166) Hostd[1050265]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 243 : Guest operation Initiate File Transfer From Guest performed on Virtual machine <VIRTUAL_MACHINE_NAME>.
2024-06-04T21:53:35.409Z In(166) Hostd[1050754]: [Originator@6876 sub=Guestsvc.GuestFileHandler] Guest File Path is <SOURCE_FULL_PATH>
Source: /vmfs/volumes/<DATASTORE_GUID>/<VM>/vmware.log (of the VM from which the file was downloaded)

2024-06-04T21:53:35.220Z In(05) vmx - VigorTransportProcessClientPayload: opID=c4c9713c seq=3576: Receiving GuestOps.InitiateFileTransferFromGuest request.
2024-06-04T21:53:35.263Z In(05) vcpu-1 - VigorTransport_ServerSendResponse opID=c4c9713c seq=3576: Completed GuestOps request with messages.
Source: hostd.log

2024-06-04T21:58:00.913Z In(166) Hostd[1050740]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 245 : Guest operation Create Temporary File performed on Virtual machine <VIRTUAL_MACHINE_NAME>.
2024-06-04T21:58:00.975Z In(166) Hostd[1050232]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 246 : Guest operation Start Program performed on Virtual machine <VIRTUAL_MACHINE_NAME>.
2024-06-04T21:58:06.170Z In(166) Hostd[1050230]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 249 : Guest operation Initiate File Transfer From Guest performed on Virtual machine <VIRTUAL_MACHINE_NAME>.
2024-06-04T21:58:06.248Z In(166) Hostd[1050750]: [Originator@6876 sub=Guestsvc.GuestFileHandler] Guest File Path is C:\Users\vagrant\AppData\Local\Temp\powerclivmware151
2024-06-04T21:58:06.364Z In(166) Hostd[1050232]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 250 : Guest operation Delete File performed on Virtual machine <VIRTUAL_MACHINE_NAME>.
Source: /vmfs/volumes/<DATASTORE_GUID>/<VM>/vmware.log (of the VM to which the file was transferred)

2024-06-04T21:58:00.759Z In(05) vmx - VigorTransportProcessClientPayload: opID=c4c97156 seq=3930: Receiving GuestOps.CreateTemporaryFile request.
2024-06-04T21:58:00.913Z In(05) vcpu-0 - VigorTransport_ServerSendResponse opID=c4c97156 seq=3930: Completed GuestOps request with messages.
2024-06-04T21:58:00.929Z In(05) vmx - VigorTransportProcessClientPayload: opID=c4c97157 seq=3936: Receiving GuestOps.StartProgram request.
2024-06-04T21:58:00.975Z In(05) vcpu-1 - VigorTransport_ServerSendResponse opID=c4c97157 seq=3936: Completed GuestOps request with messages.
2024-06-04T21:58:00.993Z In(05) vmx - VigorTransportProcessClientPayload: opID=c4c97158 seq=3942: Receiving GuestOps.ListProcesses request.
2024-06-04T21:58:01.013Z In(05) vcpu-1 - VigorTransport_ServerSendResponse opID=c4c97158 seq=3942: Completed GuestOps request with messages.
2024-06-04T21:58:06.038Z In(05) vmx - VigorTransportProcessClientPayload: opID=c4c9715a seq=3948: Receiving GuestOps.ListProcesses request.
2024-06-04T21:58:06.102Z In(05) vcpu-0 - VigorTransport_ServerSendResponse opID=c4c9715a seq=3948: Completed GuestOps request with messages.
2024-06-04T21:58:06.123Z In(05) vmx - VigorTransportProcessClientPayload: opID=c4c9715b seq=3954: Receiving GuestOps.InitiateFileTransferFromGuest request.
2024-06-04T21:58:06.169Z In(05) vcpu-0 - VigorTransport_ServerSendResponse opID=c4c9715b seq=3954: Completed GuestOps request with messages.
2024-06-04T21:58:06.319Z In(05) vmx - VigorTransportProcessClientPayload: opID=c4c9715e seq=3966: Receiving GuestOps.DeleteFile request.
2024-06-04T21:58:06.363Z In(05) vcpu-0 - VigorTransport_ServerSendResponse opID=c4c9715e seq=3966: Completed GuestOps request.

Artifacts and logs collection

The collection of ESXi logs and other artifacts can be automated through the creation of a “support bundle”. The generation of a support bundle is natively supported by ESXi and can be conducted through the ESXi web management interface ESXi Host Client or in PowerShell. In addition to the /var/run/log/ folder, a support bundle also includes the outputs of various commands: running processes and active network connections, virtual machines and datastores listing, etc.

A support bundle can be manually generated through the ESXi ESXi Host Client interface at https://<ESXI_IP | ESXI_HOSTNAME>/cgi-bin/vm-support.cgi or through the “Monitor” interface (Host -> Monitor -> Logs -> Generate a support bundle).

ESXi support bundle, for a specific ESXi or all ESXi attached to a vCenter instance, can be automatically generated and retrieved using the PowerShell module DFIR4vSphere. DFIR4vSphere also supports the collection of the vCenter instance’s VI events, and the generation of a support bundle for the vCenter appliance itself. DFIR4vSphere requires that the targeted ESXi instances are attached to a (running and reachable) vCenter instance.

Install-Module VMware.PowerCLI -Scope CurrentUser

Import-module DFIR4vSphere

Connect-VIServer <VCENTER_NAME>

# vCenter collection.

# Collects VI events (for the last X days) and generate a support bundle for the vCenter appliance itself.
# The support bundle generation can be omitted by removing the -VCBundle flag.
$EndDate = Get-Date
$StartDate = $EndDate.AddDays(<-X>)

Start-VC_Investigation -VCBundle -StartDate $StartDate -EndDate $EndDate

# For large environnement, the collection of VI events can be limited to predefined events of interest.
# The following events of interest are set (in the Start-VC_Investigation cmdlet):
#"ad.event.JoinDomainEvent", "VmFailedToSuspendEvent", "VmSuspendedEvent", "VmSuspendingEvent", "VmDasUpdateOkEvent", "VmReconfiguredEvent", "UserUnassignedFromGroup", "UserAssignedToGroup", "UserPasswordChanged", "AccountCreatedEvent", "AccountRemovedEvent", "AccountUpdatedEvent", "UserLoginSessionEvent", "RoleAddedEvent", "RoleRemovedEvent", "RoleUpdatedEvent", "TemplateUpgradeEvent", "TemplateUpgradedEvent", "PermissionAddedEvent", "PermissionUpdatedEvent", "PermissionRemovedEvent", "LocalTSMEnabledEvent", "DatastoreFileDownloadEvent", "DatastoreFileUploadEvent", "DatastoreFileDeletedEvent", "VmAcquiredMksTicketEvent", "", "", "esx.audit.ssh.enabled", "esx.audit.ssh.session.failed", "esx.audit.ssh.session.closed", "esx.audit.ssh.session.opened", "esx.audit.account.locked", "esx.audit.account.loginfailures", "esx.audit.dcui.login.passwd.changed", "esx.audit.dcui.enabled", "esx.audit.dcui.disabled", "esx.audit.lockdownmode.exceptions.changed", "", "", "esx.audit.lockdownmode.disabled", "esx.audit.lockdownmode.enabled", "com.vmware.sso.LoginSuccess", "com.vmware.sso.LoginFailure", "com.vmware.sso.Logout", "com.vmware.sso.PrincipalManagement", "com.vmware.sso.RoleManagement", "com.vmware.sso.IdentitySourceManagement", "com.vmware.sso.DomainManagement", "com.vmware.sso.ConfigurationManagement", "com.vmware.sso.CertificateManager", "com.vmware.trustmanagement.VcTrusts", "com.vmware.trustmanagement.VcIdentityProviders", "com.vmware.cis.CreateGlobalPermission", "com.vmware.cis.CreatePermission", "com.vmware.cis.RemoveGlobalPermission", "com.vmware.cis.RemovePermission", "", "", "ProfileCreatedEvent", "ProfileChangedEvent", "ProfileRemovedEvent", "ProfileAssociatedEvent", "esx.audit.esximage.vib.install.successful", "esx.audit.esximage.hostacceptance.changed", "esx.audit.esximage.vib.remove.successful"
Start-VC_Investigation -LightVIEvents -StartDate $StartDate -EndDate $EndDate

# A subset of VI events can also be retrieved specifically.
# For instance, to retrieve only the successful and unsuccessful login attempts.
# Full list of available VI Events, per vSphere version:
Start-VC_Investigation -LightVIEvents -LightVIEventTypesId "com.vmware.sso.LoginSuccess", "com.vmware.sso.LoginFailure" -StartDate $StartDate -EndDate $EndDate

# ESXi collection.

# Collect basic information (running processes, open network connections, local accounts, etc.) and generate a support bundle for every ESXi in a given cluster.
# The support bundle generation can be omitted by removing the -ESXBundle flag.
Get-VMHost -Location <CLUSTER_NAME> | Start-ESXi_Investigation -ESXBundle

# Collect basic information and generate a support bundle for the specific ESXi.
Start-ESXi_Investigation -Name <ESXI_NAME> -ESXBundle


Tags: hypervisors

View on GitHub