Performance Benefits of Adding a Second Network

Question

We transfer large media files to and from a few workstations and file servers (audio/video files, as large as 20GB each in some cases) and sometimes I get the feeling the network is bogged down as a result (directory listings can take 5-10 seconds to show up, folders are "calculating size" instead of displaying their total size, etc)

Most of the workstations and both of the servers have a second unused gigabit Ethernet port. I have heard that connecting these to another switch and setting them up as additional routes under a different subnet could be helpful, but haven't seen a recent enough article on the topic to convince me that it is worth it. (I have a spare 8-port unmanaged gigabit ethernet switch, plenty of cat5e, and precious little time)

Does anyone do anything like this or know whether it would be worth the effort?

Answer 1

Running a separate parallel network is almost definitely not the best way to tackle this. It's quite a bit of work and maintenance, and isn't likely to solve your problem. Keep in mind that it won't automatically balance the traffic over both of your networks. So for example, if all clients use the second network for file sharing, then it will be saturated and directory listings will still be slow. Your first network might still be fast, but nobody will be using it for filesharing.

This is what I would try:

• Measure the network throughput on the fileserver. If it is up near 80% of the theoretical wire speed, then you need a faster network connection. You can use the second ethernet port on your fileserver and bond the two into one network interface with twice the capacity. Your switch has to support this, and there are some limitations to how that additional capacity will actually get used in practice.
• If the network throughput is not very high, then look at other factors of the fileserver to see if they're the bottleneck such as CPU (unlikely), or RAM (more will allow it to cache more).
• The most likely reason is that the disks aren't performing quickly enough. You could look into a better RAID system, use a clustered filesystem, or just shard your data into multiple fileservers. How you do this is very dependent on your applications and OS environment.

If your switch supports SNMP, I would take the time to setup Zenoss or something similar. It will graph the utilization on each port, which will greatly improve your ability to pinpoint the bottleneck. It can also graph the vital statistics of your client and server machines.

Answer 2

If your problem is network device contention, then moving the transfers to another subnet will help. You'll have to be careful to ensure that the transfers move over the second network and not the first.

However if you are trying to do directory listings on the servers involved in the transfers of those large files, your problems might be because the drives on the computers involved are too busy to step your requests in quickly. In that case, additional network capacity on the server won't help.

Answer 3

You need to measure your current network and server utilisation. Without actual data on the utilisation of your existing resources, it is impossible to say definitively whether a performance problem exists and if using the second NICs would be useful.

Another option may be to segment the network. Plug in half the clients and one server onto one switch and plug in the switches together. Trunking ports between the most used clients/servers may also be a possibility (though not if both switches are unmanaged).

Answer 4

Disk access on the file servers themselves could be the bottleneck. If files are copied/to from them, then adding extra network capacity is unlikely to help directory listings etc. from those same servers.

Years ago (before GigE switches were cheap) I used to run two separate subnets. Each machine had two NICs, one for general and one for file sharing. This sounds similar to what you are envisaging.

As you have a spare switch, give it a try. It is possible that your current switch just cannot push data around fast enough when some ports are at their maximum transfer.

Another idea: you mention you are transferring between a few workstations/server. Depending on their physical locations and available ports, you could run Firewire cables between them and do the file transfers that way (from what I remember, you can make it look like a network interface on the machines). Your disks may still be the bottleneck then, but at least you have solved the network speed issues.

Do report back what you find.

Answer 5

One thing you can do with the separate subnet (assuming it's all gigabit) is use jumbo frames for additional throughput. For this to work, all devices (including the switch) must support jumbo frames.

I have a "backend" network set up this way; the servers and NAS box each have an interface on this network (set for jumbo frames), the servers and NAS box talk to each other over this network freeing up the "frontend" interfaces for the clients.

Answer 6

If possible you could always setup VLANs if your current infrastructure supports it and you wouldn't need to run an additional network cable to each machine. But as David said above if it isn't a network issue, It won't solve anything.