Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am testing a storage appliance and need to write TeraBytes of data to it. Using fio or dd takes days to write that much amount of data. Are there any (free or not-free) tools/applications, that I could use for this purpose. The tool should generate the data and write it, fast. Think, I need to use a hadoop database for this purpose..what I am not sure about here, is, what would generate that data (that hadoop would then write)?

share|improve this question

closed as off-topic by joeqwerty, Dave M, mdpc, Ward, squillman Sep 27 '13 at 1:58

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions seeking product, service, or learning material recommendations are off-topic because they tend to become obsolete quickly. Instead, describe your situation and the specific problem you're trying to solve." – joeqwerty, Dave M, mdpc, Ward, squillman
If this question can be reworded to fit the rules in the help center, please edit the question.

1  
I'm confused as to how dd if=/dev/zero of=/dev/yourdisk bs=stripesize could be slower than another program that "generates data fast", It seems like that would be bound to your disk's io bandwidth. How are you running dd or fio? –  Andrew Domaszek Sep 25 '13 at 18:25
    
Unless you're doing something horribly wrong/weird dd is going to generate data as fast as the slowest device that you're reading from/writing to. There is not tool on earth, no matter how expensive, that is going to overcome the physical limitations of your hardware. –  Sammitch Sep 25 '13 at 18:30
    
What is your performance goal? Hadoop uses multiple hosts for both storage and job processing. Is almost reads like you are saying that a single instance of dd from your test host is not fast enough. If that is the problem you may need to connect additional hosts to your network, and test with multiple I/O streams. –  Rik Schneider Sep 25 '13 at 19:01
    
Is /dev/urandom going to be any slower than /dev/zero? I was using that, for dd. And I will try larger block size.. –  user2816476 Sep 25 '13 at 19:34

1 Answer 1

You stated that dd is to slow for you. How are you using it? Because dding /dev/zero into some file seems fast to me, also you can reduce writes to disk (and thus effectively speed up) the dd with its obs parameter.

You probably should first find, where the bottleneck of generating/writing data is.

share|improve this answer
    
Is /dev/urandom going to be any slower than /dev/zero? I was using that, for dd. And I will try larger block size.. –  user2816476 Sep 25 '13 at 19:30
    
It is slower. Zero is just a stream of 0x00 but the urandom is random data generator. The generation of data cost some computing power. IMHO the bottleneck here is probably disk I/O and not the data source. –  Fiisch Sep 25 '13 at 20:07

Not the answer you're looking for? Browse other questions tagged or ask your own question.