As I see it, the question you are asking is
Is there any reason to create a separate vSwitch and/or network for communication between VMs on a single host?
When I have questions like this, I always start off by asking myself the following:
What would I do in this situation if there were no virtualization?
To me, the analog to your environment for a non-virtualization scenario is: some systems plugged into the same "mostly dumb switch", some of them only talk to each other and some talk to everyone else, including the world. So the question could be:
Is there any reason to use a separate switch for communication between systems that only need to communicate with each other (or direct connection, if there are only two)?
Of course, the answer to this question is basically the same as the answer to all good questions: it depends. It depends on a lot of different factors, that to me fall into the following categories:
- Privacy (not really the same as security)
- Ease of troubleshooting
So, let's take each category on its own:
Unless you are pushing max speeds on many of the connected systems, or have lots of "cpu intensive" traffic, like multicast, then probably not.
Do any of the machines have critically important stuff on them? Are any of the "private" machines vulnerable to a malicious attack from one of the other systems or even from outside the network?
Should the traffic between the hosts that communicate among themselves be hidden from other systems on the network? Is there any chance that a system put into promiscuous mode could listen in on the other traffic (this is where the "mostly dumb switch" that is a stock vSwitch comes into play: it can be configured to allow hosts to enable promiscuous mode). Also, keep in mind that all of the systems will use some amount of broadcast traffic that the others will see, even if it is only arp.
Does adding complexity increase the reliability of the system or reduce it? If a[|the] switch fails or is accidentally misconfigured, are you ok with it taking down all of the communication between systems?
5. Ease of troubleshooting
If something goes wrong and you need to troubleshoot, will you be able to easily isolate the different systems?
Can you explain the setup to someone else easily? Will the big picture be growing or changing over time? If you are like me: will you remember how it is set up in six months, a year, two years, five years? Can you figure out how it is set up at a glance? If you have to move the whole thing, or relocate some of the systems, will it be easy, or hard?
Does any of the above actually matter? Are you able to realistically implement any of the above? If it is a lab that you are going to tear down in a couple of weeks, does anything other than performance matter? If you don't know how to configure additional vSwitches or can't get an additional physical switch in the "non-virtualization analog", then does it make a difference?
You'll have to weigh the pros and cons of each scenario yourself and implement it as you see fit, or provide WAY more information to us so we can make a recommendation.
But, based on the information I have from your post so far, if I were in your shoes, I'd separate them. I'd do this even if it only gets me clear boundaries and lines of communication between systems, and so I can understand what is and isn't going on. I'd configure a new vSwitch for the "private" systems, not connected to any physical NICs on the host, and rename the port group in it to something like "private network for host-to-host only" (I can't remember if the name field will accept that many characters, and I'm too lazy to fire up vSphere just to check, sorry). This will also make it easier in the future if you end up growing to multiple hosts.