Yeah my VM connects directly to our TV and I have a dedicated USB
controller passed through to windows that I plug a wireless
keyboard/trackpad into. I also do disable the ESXi Vga card. I haven't
had an issue with reverting back, anytime I unplug my monitor from the
physical video card it turns the other one back on and I can get to it
via the esxi console.