I am testing a storage appliance and need to write TeraBytes of data to it. Using fio or dd takes days to write that much amount of data. Are there any (free or not-free) tools/applications, that I could use for this purpose. The tool should generate the data and write it, fast. Think, I need to use a hadoop database for this purpose..what I am not sure about here, is, what would generate that data (that hadoop would then write)?
1 Answer
You stated that dd is to slow for you. How are you using it? Because dding /dev/zero
into some file seems fast to me, also you can reduce writes to disk (and thus effectively speed up) the dd with its obs
parameter.
You probably should first find, where the bottleneck of generating/writing data is.
-
Is /dev/urandom going to be any slower than /dev/zero? I was using that, for dd. And I will try larger block size.. Sep 25, 2013 at 19:30
-
It is slower. Zero is just a stream of 0x00 but the urandom is random data generator. The generation of data cost some computing power. IMHO the bottleneck here is probably disk I/O and not the data source.– FiischSep 25, 2013 at 20:07
dd if=/dev/zero of=/dev/yourdisk bs=stripesize
could be slower than another program that "generates data fast", It seems like that would be bound to your disk's io bandwidth. How are you running dd or fio?dd
is going to generate data as fast as the slowest device that you're reading from/writing to. There is not tool on earth, no matter how expensive, that is going to overcome the physical limitations of your hardware.