I agree with the other posts that it could be a bandwidth issue.
Here are some things to check:
Can you continually ping the hop before your server.
Run a 'tracert your.server.name.com' and get the last pingable hop before your server. Continually ping this host along with your server. Does it respond the same? Try the hop before that as well. This will indicate where the start of the network issues may occur. If it is only your server that is dropping pings then that would be a good conclusion that it is not other network equipment.
Start with the event log, check for errors in the application and system log.
It could be that your network card is throwing errors. It may just need a driver update, check with your vendor. You may notice a reoccurring event that is also happening when your server is "overloaded"
Ping the server from another server on the same network switch.
If the ping from a server on the same LAN are ok, then yes it is leading back to being a bandwidth issue, if not, then it sounds like it is a local LAN networking issue which could be anything from duplex mismatch to incorrect switch configuration. Try and eliminate each item piece by piece.
Get some monitoring in place
If possible get some monitoring up for your server, nagios, perfmon, zenoss, opsview, cacti, ntop, whatever your flavor the more information you have about what is going on inside and outside of your server, the better idea you will have about what is going wrong.
Dig deeper into IIS and application logs
Perhaps a big file or long running process is choking your server, it doesn't sound like it from your post it sounds like cpu/mem are low, but also think about disk and everything else. Check the IIS logs and log what you can from your db/website if possible.
Get statistics for what your site is doing.
Take a look at your available bandwidth, and how much data you are serving. Are you on a 10mbit line, and simply maxing it out with large downloads/uploads? Remember if you completely max your upload this will also effect your download, and could even interrupt your packets.
Hope this helps you, the more information you can provide the better answers you should get.