I'd always start with two Microsoft SysInternals tools: Process Explorer and Process Monitor. Plus, in this instance, I'd add Disk View.
Using these you can:
Process Explorer : View CPU, memory and I/O stats (more clearly than the built-in tools)
Process Monitor : Actually watch what the machine is doing. One of the default filters is to drop anything from the "System" process. Turn this filter off when diagnosing a file server issue. However, do turn on "drop filtered events".
Disk View : Watch disk load.
Being a file server, I'm guessing you have Anti-Virus on-access scanning enabled? Plus, you may have volume shadow copy snapshots running? Could you confirm whether this is the case?
Generally, with x64 file servers, I'd recommend setting the page file to "system managed" and keep it on a separate volume. Plus I'd keep a close eye on the "peak commit charge". Having monitored this for a while, you should be able to judge whether more memory is required.
Also, can you post details of the clients (i.e: what os version/edition)?
I've seen issues with XP clients running against Win2008R2. When I remember the specifics, and more importantly, the solution, I'll update this post...