It sounds like this is WORKING AS DESIGNED. You probably have over-committed the amount of physical memory in your host/cluster and VMWare is using its balloon driver to make the host think it is low on memory so that it frees up memory marked for garbage collection so the ESXi hypervisor can know to then use those memory locations to provide memory to other VMs that need it.
The below blog post is excellent at explaining what is going on. The workaround is to set a reservation for your host:
http://www.vfrank.org/2013/09/18/understanding-vmware-ballooning/