In "Configure a DNS Server to Use Forwarders", Microsoft offers this little nugget of wisdom:
You can prevent common problems that are associated with forwarders by configuring your DNS servers to avoid overusing your forwarders.
While that's primarily advice in the context of conditional forwarders, it applies generally as well. Forwarders cannot magically determine a DNS server is unavailable: it has to try asking. Sure, they could so some sort of UDP 53 portscan, but that just adds complexity, latency, and a chance of triggering some overzealous IPS. Instead of all that, it simply performs a DNS query, which only takes a few seconds. If you wish to immediately alleviate this issue, you could set a much more aggressive timeout (say, 1 second) and perhaps add a few forwarders. However, this runs clear risks that you won't get any responses at all.
You may want to consider configuring your DNS to point to 2 or more "Edge DNS" servers, which themselves recurse to root hints (recursion to root hints is not some grand evil; it really doesn't take that long. Generally, that's going to be shorter than the 3 seconds you're facing now.) Or, perhaps you may want to consider recursing to root hints directly and not using forwarders at all.
I assume your concern here is external resources for which you are not an authority. In that case, the above are two starting paths to consider; there are others as well of course. If the case is an authoritative or internal zone, then consider a zone design which takes advantage of better features than a server-level forwarder to achieve the answer, such as Stub Zones.