This problem is still driving us crazy. At the time of these screenshots, I am down to 17 VMs on a machine with 40 cores (with Hyper-threading 80) and 512GB of RAM. Total vCPUs is less than 34. Total vRAM ~ 120GB.
Highlighted line on this graph is CPU Latency. Actually pegged at 100% for almost 10 minutes. Entire time is during high disk I/O as I was cloning a VM from one datastore on this virtual SAN to another.
As you can see by this graph and table below it, vCPU 0 maximum usage is 431 Mhz, but the CPU Latency was pegged at 100% for almost 10 minutes. And since Usage includes this Latency number, I am getting CPU alarms right and left on my VMs. As you can see from earlier posts, the cpu usage of the vmx process includes CPU Latency on that process and Total Usage of the VM is a total of usage on all the worlds.
As stated earlier, %Lat_C is coming from the vmx process. Actual vCPU utilization maximum is about 400 Mhz. Yet performance of this virtual SAN is terrible.
N%L is 100%, so vCPUs are not moving around, all data is local to the NUMA node. There is no reason for CPU Latency in this environment. Every vCPU has it's own core, let alone Thread.