We just had a very similar issue on our system. We are running 5.1 ESXi and 5.5 vCenter.
We added a new 1.5 TB datastore to our VMware cluster. Only a few hosts were seeing the new datastore. After doing a bunch of refresh/rescan all we managed to get all the VMhosts to see the new datastore. Then things got really weird, the size was being reported erroneously from it being 4TB to over 290,000TB!!! It would jump into a red alarm and then disconnect from some hosts. On other hosts it showed up correctly as 1.5TB but with 700GB used! Refresh again and the datastore would disappear!
I checked on our SAN and there were no errors or messages to indicate any issue with the SAN, LUN, or communication paths.
We ended up creating a new LUN and datastore and just deleted the one causing the issues. So far everything seems fine. We have never seen this happen before. We too concluded that it may have been some type of corruption on the LUN or during the VMFS5 formatting. I know this doesn't offer much help in how to solve the issue. For us since it was a new datastore with nothing on it we opted to just delete it and move on rather than troubleshoot deeper with VMware.