So, thinking the driver version might have something to do with this issue, I referred to this page:
VMware Compatibility Guide: I/O Device Search
So I thought, great, ESXi 4.0 U2 has the same driver version, so I can try it. Well, I did, and it didn't work either. I have also verified that this card works perfectly under Ubuntu 12.0.4, so something has changed with ESXi's network stack and how it handles this card between 3.5 and 4.0.