If IPv6 is misconfigured on your server, then it not only could be part of the problem, but it probably isn't serving anybody anyhow. A scan of the web server logs or a bit of network monitoring should show how many endusers, if any, are using IPv6 to reach you. If there seem to be few or none, then the diagnostic benefit of running without it for a trial period likely outweighs the cost. And if that fixes it, then conclude that there's some misconfiguration in your ipv6 setup that ought to be addressed.
I've had direct involvement with two incidents in which misconfigured ipv6 caused very significant performance issues. In both cases (and I'm going to go light on details) IPv6 appeared to be running, so applications attempted to use it, but it wasn't being properly routed, so the traffic timed out. A timeout for, say, a dns lookup on every mail message or every web request will kill performance for sure.
Swearing off IPv6 completely is a bit Luddite. The moral is: if it's enabled, make sure it's properly configured.