Could be related to your CRL(*). AFAIK chrome does checks CRL while Firefox does not, and I don't know about IE.
Try with "check for server certificate revocation" enabled and disabled. I have no idea though how often chrome checks the CRL lists.
That's what I can think of. Could be something else as well.
(*) For those who happen to be new to SSL: CRL (Certificate Revocation List) - normally SSL certificate stores an address where it's possible to check if this certificate is not revoked. There's two protocols for such check - one is OSCP another one is CRL. Browsers once encounter SSL certificate may check if it isn't revoked (if they actually do it or not depends on browser and your settings). This check often disabled, but when it is enabled, browser may first contact revocation server and only then load the web page.