For any new server, take time to work out what happens during normal use. Only then will you be able to identify what's not right.
Most server-affecting issues are visible on the Performance tab, so start there and then drill down. Alternatively, start at the top of the Application and System event logs. Until you become familiar with them, you may have to Google these to determine if they're extraordinary.
As a general rule, most failing processes will not be in
dllhost.exe (which hosts IIS objects). It's better to trace the problem than guess it from the process, so map network usage and locked files to processes using Process Explorer. If that's impossible, become familiar with these:
These have the advantage of working over telnet/psexec. If you find something has gone mad, try to use the services tab to stop the related service before killing the process. And never close handles, no matter how tempting it is.
Also, don't neglect Performance Monitor. It takes a little more work than other sources, but will allow you to start logging candidates if you missed the spike.