The licenses that are there for 4.x will not work with 5.x. You will first need to upgrade the licenses to newer version by logging into your MyVMware portal.
There is a KB Article which explains the procedure to do this. Please follow it and upgrade your keys VMware KB: Upgrading license keys in My VMware
Have you checked if you have enough licenses to apply for all hosts? That might be the issue.
For hosts reporting that they are still being managed by other vCenter, You will have to first disconnect the host and then remove the host from vcenter inventory. This will uninstall the vcenter agent on the ESXi host.
Once the agent is uninstalled you wont get a message telling its managed by another vcenter again.
Let me know if you need more help.
Message was edited by: Abhilash