Have you checked all of your VM's to ensure that none of them are still using the swap volume?
If a VM was powered on, on the host that was configured with swap vol, then it will still be using that swap volume. You need to make sure it's been disabled on all hosts and all VM's are moved off.
If ANY host was using the swap vol, then you will need to move those VM's and also reboot them. Have you tried unmounting the datastore from the esxcli at all? What error does that give if you look in the vmkernel log?