I realize I'm coming back to this question much later than when it was posted, but I had forgotten that I posted this question and when I saw it, I wanted to share the solution with others.
It turned out the problem was my monitoring solution of all things. I had a check setup for the system to check the https page for the host every 5 minutes which for some reason would eventually cause the system to respond to everything, to the point where vSphere clients couldn't even connect anymore.
I disabled this check (relying instead on pings) and this problem hasn't returned for almost a year now.
I'm guessing there is a security setting somewhere under the hood of ESXi 4.x that tells the system to stop responding after a specified number of requests, but I wasn't able to locate that.
When the issue would start occurring, the VMs would stay up, however you couldn't connect to anything at the hypervisor level until completely rebooting the system (even restarting the management services would not fix it).
I tried updating to the most recent versions of ESXi but that did not resolve the issue.