Splunk Guide: Detect Suspicious Activity Before ESXi Ransomware

) Data you must onboard (minimum)

ESXi host syslog → forward /var/log/* (notably shell.log, hostd.log, syslog.log, auth.log, ssh.log) to Splunk. Use Splunk Add-on for VMware ESXi Logs (Splunk_TA_esxilogs). Splunk Documentation+1
ESXi log paths & what they contain (for validation): shell.log = interactive shell commands; auth.log = local logins; ssh.log = SSH events; hostd.log = VM power/management ops. Support Portal Nakivo vStack
Optional but recommended: vCenter logs (audit, plugin access) via the Splunk Add-on for vCenter Logs. Splunk Documentation
Transport tip: prefer TCP/SSL; avoid UDP behind load balancers (can corrupt/ reorder vSphere syslog). splunk.github.io

Why now? ESXi ransomware waves (e.g., ESXiArgs) abused exposed/ unpatched vCenter & ESXi to gain code execution, then used shell/SSH to mass stop VMs and encrypt configs. CISA VMware varonis.com

2) High-signal precursor behaviors (what to hunt)

TTP themes: enable shell/SSH, new accounts, persistence (/etc/rc.local.d/local.sh), bulk VM operations, exploitation of old vCenter plugin paths (CVE-2021-21972/21985), use of vim-cmd/esxcli. Tenable®+1 Rapid7 Support Portal

3) Ready-to-use SPL (adjust `index=` and `sourcetype=` to your environment)

Tip: If you use the ESXi Logs TA defaults, you’ll typically match a syslog sourcetype (e.g., vmware:vsphere:esx:syslog / vmw-syslog). Validate with | tstats count where index=<your_index> by sourcetype. Splunk Documentation

A. ESXi Shell / SSH enabled (DCUI or API), or first use

index=<esxi_index> (sourcetype="vmware:vsphere:esx:syslog" OR sourcetype="vmw-syslog" OR sourcetype="syslog")
("ESXi Shell" OR "TSM-SSH" OR "SSH") (enabled OR started OR "Start and stop with host")
| rex field=_raw "(?<user>\broot\b|\w+)"
| stats earliest(_time) as first_seen latest(_time) as last_seen values(user) as users by host

(Watch for sudden SSH/ Shell starts on production hosts.) techdocs.broadcom.com

B. Interactive shell commands (look for `vim-cmd`/`esxcli`)

index=<esxi_index> source="/var/log/shell.log"
| rex field=_raw "(?<cmd>(vim-cmd|esxcli)\s+[^\r\n]+)"
| timechart span=5m count by host

Spike of vim-cmd vmsvc/*, hostsvc/enable_ssh, maintenance_mode_enter, esxcli network firewall set -e false is suspicious. Nakivo

C. Mass VM power/ unregister operations in short window

index=<esxi_index> source="/var/log/hostd.log"
("PowerOffVM" OR "PowerOnVM" OR "UnregisterVM" OR "DestroyVM" OR "UnregisterVM_Task")
| bin _time span=5m
| stats count values(_raw) as samples by host _time
| where count>=10

(High burst often precedes encryption.) vStack

D. New/changed local accounts or root password change attempts

index=<esxi_index> (source="/var/log/shell.log" OR source="/var/log/syslog.log")
("esxcli system account add" OR "esxcli system account set" OR "passwd" OR "chpasswd")
| rex field=_raw "(?<account>(?<=account add --id\s)\S+|(?<=for user\s)\S+)"
| stats earliest(_time) as first_seen values(account) by host

(Attackers often add users for persistence.)

E. Persistence via boot script or SSH keys

index=<esxi_index> (source="/var/log/shell.log" OR source="/var/log/syslog.log")
("/etc/rc.local.d/local.sh" OR "rc.local" OR "/etc/ssh/keys-" OR "authorized_keys")
| stats values(_raw) as evidence by host

(Unauthorized edits to local.sh persist commands across reboots; SSH keys cement access.) Support Portal

F. ESXi firewall tampering or syslog changes

index=<esxi_index> source="/var/log/shell.log"
("esxcli network firewall set" OR "esxcli network firewall ruleset set" OR "esxcli system syslog config set")
| stats values(_raw) as evidence by host

G. vCenter exploitation probes for legacy plugin paths (outside ESXi but relevant precursor)

index=<vcenter_index> (sourcetype=wineventlog:security OR sourcetype="iis:log" OR sourcetype="vcenter:access" OR sourcetype="http:access")
(uri_path="/ui/vropspluginui/rest/services/uploadova" OR uri_path="*/vropspluginui/*" OR uri="/ui/vropspluginui*")
| stats count by src_ip, uri_path, status, host

(Indicators tied to CVE-2021-21972; add vSAN plugin checks for CVE-2021-21985.) Tenable®+1 Rapid7

H. ESXiArgs-style operator behavior (quick sweep)

index=<esxi_index> (source="/var/log/shell.log" OR source="/var/log/hostd.log")
("getallvms" OR "vmsvc/power.off" OR "rm -rf *.vmx" OR "find /vmfs/volumes" OR "tar -cf" OR "openssl")
| stats count values(_raw) as samples by host

(Adapt to your observed actor TTPs; see ESXiArgs advisories for context.) CISA VMware

4) Correlation searches (Enterprise Security friendly)

CS-1: “ESXi Pre-Ransomware—Shell + Bulk VM Ops”

Logic: A + C within 30 minutes on same host.
Risk=90; Urgency=High; MITRE: TA0002/TA0003/TA0040.

CS-2: “ESXi Persistence—local.sh or SSH Keys Modified”

Logic: E AND (A OR B) within 2 hours. Risk=80.

CS-3: “External Exposure—vCenter Exploit + New ESXi User”

Logic: G (from internet IP) AND D on any ESXi behind that vCenter within 60 minutes. Risk=95.

5) Dashboards (one-page triage)

Panel 1: SSH/Shell state timeline per host (search A).
Panel 2: Top commands (vim-cmd, esxcli) last 24h (search B).
Panel 3: Burst VM power/unregister (search C).
Panel 4: Persistence edits (search E).
Panel 5: vCenter exploit URIs (search G).

6) Response playbook (quick)

Isolate host mgmt (firewall block SSH/HTTPS to mgmt VLAN except admin jump hosts).
Revoke persistence (revert local.sh, remove rogue accounts/keys). Support Portal
Patch & close exposure (prioritize KEV/vCenter plugin CVEs; verify no internet-exposed vCenter/ESXi). CISA Tenable®+1
Review logs for staging (bulk vim-cmd & VM ops, odd backups/archives). Nakivo vStack
Hunt for ESXiArgs IOCs in historical data. CISA cloudsek.com

7) Hardening checklist (prevention boosts)

Disable/limit Shell & SSH; use DCUI only when needed; alert on state changes. techdocs.broadcom.com
Aggressive patching of vCenter/ESXi; CVE-2021-21972/21985/21974 were heavily abused in past ESXi waves. Tenable®+1 varonis.com
No external exposure of management interfaces; restrict by firewall/VPN; send syslog to Splunk TA. Splunk Documentation

Cyberdudebivash