Since converting online backup to an in-house backup (ArcServe) performance on this server has degraded considerably. Does anybody have recommendations on tuning for performance?
You need to determine what is actually performing poorly, usually via symptoms and measuring/monitoring, diagnose the cause, and then tune that.
Define what you mean by "performance has degraded considerably." Does the server only do backups, and now they take longer? Or does it also serve files, and users are complaining that it's slow to do so?
Start with Task Manager (while the problem is happening) - look at the CPU and memory. Are they seeing high utilization? If so, what's using the most of what's being hammered, in the process list? That should give you some pointers.
To get long-term trending of this, use Perfmon, or enable SNMP and use an SNMP collector with graphing, like MRTG/RRDTool/Cacti, or even a modestly-priced app like WhatsUp or SolarWinds Orion if open source (which sometimes can mean "runs best or is better supported on Linux) isn't something that you're comfortable with.
Sometimes, you have to dig deeper. It might not just be low free RAM or high CPU utilization. You could be running out of open file handles, or page table entries, or paged pool. Those can be trickier to monitor and diagnose, but you'll need to read up on them to know the symptoms. Things like this will often give some clues in the event log of the server. Have you checked there?
Has anything else changed on this server? Did you install a tape drive on it? Is it terminated correctly (if SCSI)? Is it attached to the same adapter as any of the hard drives for this system? There could be something sub-optimal that you introduced when you added the tape drive.
Engage the vendor. Did you buy support with ArcServe? Call them and explain the problem. If you didn't buy support, you do probably have access to their knowledgebase, they may even have forums on their site.