It is the pink PSOD screen. This article explains what is happening. My understanding is if you disable SMT in the bios you can run ESXi 6.5. But what is the point in that? By disabling SMT you are turning off "Hyper Threading". So you go from a great 16 thread CPU back down to 8 threads.
AMD Ryzen "Working" with VMware ESXi 6.5 to an extent
I check the site weakly for an update but nothing since April. AMD also has threadripper on the way and then their server line of CPUs. So I expect VMware will eventually get this worked out. So for now I have a 1800X sitting on the shelf waiting for the update.