Increasing the port count on the switch will take away from the remaining port count on the host, which is 4096 in vSphere 5.5, or on the switch (60,000 for a vDS). Depending on the version of ESXi, it may take a reboot. Based on your 4 host scenario...no penalty that will affect your environment.
You should always have more ports available than you will need, to ensure you don't run out unexpectedly.