nas question regarding TeraByte size vs cpu power for basic storage of data

Question

is there a relationship between storage space and number of cpu's required to handle it? if i have 100GB of data a night coming in through a fat pipe remotely from 50 different sites so that the total data footprint over a 18hr period is 100GB, and i wanted to have a 20TB NAS or storage system receiving it, does the NAS server need to have, for example, 1 xeon or should it be a dual cpu with multi core etc

the purpose of the NAS is redundancy for data in the field. Assume that there is no bandwidth issue and we will be using fat pipes to move the data, my concern is the simultaneous piping of seperate geographic instances of 2GB of data coming in from 50 - 100 seperate sites. i don't want to have read/write issues or missing/crashing of a sync.

i need a direction to start testing with this idea, so if i have to move that much data onto a NAS server nightly.

if it is too vague a question i can elaborate it more, thanks.

Answer 1

The processing power required to handle storage depends on many factors. These days storage processing is a distributed process. But the various places where processing is encountered:

• The file-sharing protocol. SMB, AFP, NFS, and iSCSI all have their own impacts to loading. Code-quality has a significant impact as well; a Mac can scale AFP w-a-y farther than a Linux-based NAS using the netatalk package for AFP support. 3.0 Samba behaves worse than 3.6 Samba.
• The kernel I/O routines. Some kernels handle I/O more efficiently than others. A 2.6.4-era Linux kernel doesn't behave as well as a 2.6.28, or 2.6.36.
• Software vs. Hardware RAID. If using a software parity raid (R5 or R6) the CPU involved in I/O operations can get significant. But if you're striping (RAID1 or RAID10, or strongly-not-recommended RAID0) it barely twitches the needle. If you're using hardware RAID, they generally scale pretty far before hitting limits, but at that point your CPU loading doesn't matter much.
• Network stack. Crappy NIC drivers can cause CPU spikes. Hard to predict.

All of the above provide the processing multiple applied to incoming storage requests. Doing 6.2 MB/s of pure write I/O will put a certain amount of load, but the range of loading experienced by a system can go from negligible to crushing depending on how all of the above factors shake out.

For instance, Windows Server 2008R2 on actual server hardware with a couple of disks on a hardware RAID controller can take that 6.2 MB/s all day and night and barely break a sweat (baring bad drivers), even on relatively old 64-bit Pentium 4 CPUs. A FreeNAS based on a Core2 processor doing the same write rate over AFP may not be able to keep up.

Which is a long way of saying that terabytes stored is a very poor predictor of CPU loading.

Answer 2

The relation would be not as much a pure space:cpu issue. Depending on the solution you are aiming at, you might need CPU power for features like deduplication, compression, encryption or hash / checksum calculations.

A simple data copy process would not incur much CPU overhead unless you have a really badly designed system. Your requirement of 100 GB per 18 hours would average to 1,58 MB/second - even a netbook would be able to cope with that nowdays.

You also should concentrate on the I/O backend you are using. While writing 1,58 MB/second does not sound terribly challenging, using 50-100 simultaneous processes to do so will incur a lot of random write load. Hard disks do not cope well with random (write) loads as these will incur a lot of time-intensive head seeks, so you will need to have something that cushions randomness - like a DRAM or SSD write cache.