-2

I am testing a storage appliance and need to write TeraBytes of data to it. Using fio or dd takes days to write that much amount of data. Are there any (free or not-free) tools/applications, that I could use for this purpose. The tool should generate the data and write it, fast. Think, I need to use a hadoop database for this purpose..what I am not sure about here, is, what would generate that data (that hadoop would then write)?

4
  • 1
    I'm confused as to how dd if=/dev/zero of=/dev/yourdisk bs=stripesize could be slower than another program that "generates data fast", It seems like that would be bound to your disk's io bandwidth. How are you running dd or fio? Sep 25, 2013 at 18:25
  • Unless you're doing something horribly wrong/weird dd is going to generate data as fast as the slowest device that you're reading from/writing to. There is not tool on earth, no matter how expensive, that is going to overcome the physical limitations of your hardware.
    – Sammitch
    Sep 25, 2013 at 18:30
  • What is your performance goal? Hadoop uses multiple hosts for both storage and job processing. Is almost reads like you are saying that a single instance of dd from your test host is not fast enough. If that is the problem you may need to connect additional hosts to your network, and test with multiple I/O streams. Sep 25, 2013 at 19:01
  • Is /dev/urandom going to be any slower than /dev/zero? I was using that, for dd. And I will try larger block size.. Sep 25, 2013 at 19:34

1 Answer 1

2

You stated that dd is to slow for you. How are you using it? Because dding /dev/zero into some file seems fast to me, also you can reduce writes to disk (and thus effectively speed up) the dd with its obs parameter.

You probably should first find, where the bottleneck of generating/writing data is.

2
  • Is /dev/urandom going to be any slower than /dev/zero? I was using that, for dd. And I will try larger block size.. Sep 25, 2013 at 19:30
  • It is slower. Zero is just a stream of 0x00 but the urandom is random data generator. The generation of data cost some computing power. IMHO the bottleneck here is probably disk I/O and not the data source.
    – Fiisch
    Sep 25, 2013 at 20:07

Not the answer you're looking for? Browse other questions tagged .