This is most likely caused by the VMFS block size for the target datastore. In order to allow files larger than 512GB the block size on a VMFS3 datastore needs to be at least 4MB. Do you have another datastore with a 4MB (or 8MB) block size? If so, create an appropriately named folder on that datastore for the .vmdk file, and run vmkfstools using this datastore/folder as the target.
André