Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

is there a relationship between storage space and number of cpu's required to handle it? if i have 100GB of data a night coming in through a fat pipe remotely from 50 different sites so that the total data footprint over a 18hr period is 100GB, and i wanted to have a 20TB NAS or storage system receiving it, does the NAS server need to have, for example, 1 xeon or should it be a dual cpu with multi core etc

the purpose of the NAS is redundancy for data in the field. Assume that there is no bandwidth issue and we will be using fat pipes to move the data, my concern is the simultaneous piping of seperate geographic instances of 2GB of data coming in from 50 - 100 seperate sites. i don't want to have read/write issues or missing/crashing of a sync.

i need a direction to start testing with this idea, so if i have to move that much data onto a NAS server nightly.

if it is too vague a question i can elaborate it more, thanks.

share|improve this question

2 Answers 2

up vote 0 down vote accepted

The processing power required to handle storage depends on many factors. These days storage processing is a distributed process. But the various places where processing is encountered:

  • The file-sharing protocol. SMB, AFP, NFS, and iSCSI all have their own impacts to loading. Code-quality has a significant impact as well; a Mac can scale AFP w-a-y farther than a Linux-based NAS using the netatalk package for AFP support. 3.0 Samba behaves worse than 3.6 Samba.
  • The kernel I/O routines. Some kernels handle I/O more efficiently than others. A 2.6.4-era Linux kernel doesn't behave as well as a 2.6.28, or 2.6.36.
  • Software vs. Hardware RAID. If using a software parity raid (R5 or R6) the CPU involved in I/O operations can get significant. But if you're striping (RAID1 or RAID10, or strongly-not-recommended RAID0) it barely twitches the needle. If you're using hardware RAID, they generally scale pretty far before hitting limits, but at that point your CPU loading doesn't matter much.
  • Network stack. Crappy NIC drivers can cause CPU spikes. Hard to predict.

All of the above provide the processing multiple applied to incoming storage requests. Doing 6.2 MB/s of pure write I/O will put a certain amount of load, but the range of loading experienced by a system can go from negligible to crushing depending on how all of the above factors shake out.

For instance, Windows Server 2008R2 on actual server hardware with a couple of disks on a hardware RAID controller can take that 6.2 MB/s all day and night and barely break a sweat (baring bad drivers), even on relatively old 64-bit Pentium 4 CPUs. A FreeNAS based on a Core2 processor doing the same write rate over AFP may not be able to keep up.

Which is a long way of saying that terabytes stored is a very poor predictor of CPU loading.

share|improve this answer

The relation would be not as much a pure space:cpu issue. Depending on the solution you are aiming at, you might need CPU power for features like deduplication, compression, encryption or hash / checksum calculations.

A simple data copy process would not incur much CPU overhead unless you have a really badly designed system. Your requirement of 100 GB per 18 hours would average to 1,58 MB/second - even a netbook would be able to cope with that nowdays.

You also should concentrate on the I/O backend you are using. While writing 1,58 MB/second does not sound terribly challenging, using 50-100 simultaneous processes to do so will incur a lot of random write load. Hard disks do not cope well with random (write) loads as these will incur a lot of time-intensive head seeks, so you will need to have something that cushions randomness - like a DRAM or SSD write cache.

share|improve this answer
    
thanks for the reply, can you expand or describe the ssd write cache? thanks. –  user8256 Apr 22 '11 at 15:00
    
Sun/Oracle has something called "Hybrid storage pools" where fast random-access devices (SSDs) can be used either as a read cache (L2ARC) or persistent and transaction-secure write cache (ZIL). There is an oldish solution brief which gives a rough overview but is bare of technical details: download.intel.com/design/flash/NAND/… –  the-wabbit Apr 26 '11 at 22:06

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.