I need to encrypt a 200gb file.
Using 7z, on a pretty fast win2012 server, it is going to take 12 hours (using the fastest option).
Is there a faster way?
asked Mar 29, 2016 at 3:46
Jonesome Reinstate MonicaJonesome Reinstate Monica
5,39510 gold badges55 silver badges82 bronze badges
3 Answers
I have found using the 7z command line to be faster than the gui for large files.
A suggestion could be to store the file on an encrypted virtual drive image instead of using 7z.
Alot of this is going to be:
- Disk I/O - because you probably don't have 200GB of RAM, it's going to have to load blocks in at a time and then write them out. So, to improve this, use a high-speed (6GB/s) SSD on SATA3, or a RAID card for even faster speeds.
- Block size - This may not work as well on Windows, but on OSs where you can choose or tune the filesystem, managing the block read/write sizes relative to the bus and RAM can improve this considerably.
- RAM - The more you can do in RAM, the faster it gets, and RAM speed has an effect also, particularly if your CPU can dump unattended I/O directly between RAM and disk.
- CPU - CPUs with crypto-related maths extensions can do better for that crypto type (e.g. AES-NI), particularly if they can access the data directly rather than via a pipe.
answered Mar 29, 2016 at 4:52
-
If a 200GB file is taking 12 hours this is not I/O-bound. It's also unlikely to be memory-bound although the evidence is not so direct. In practice this is a rare case where a problem really is compute-bound only. So don't spend lots of money on SSD or memory to improve performance here.– user337366Mar 29, 2016 at 6:31
-
This doesn't answer the question well. The question includes vital details that weren't taken into account in this answer:
7zis a file archiving and compression program andfastestis a method of low-quality but quick compression. Compression is very slow compared to encryption. The compression can be turned off to greatly speed things up, as shown in @samsmith's answer to their own question, where the time went down from 12 hours to 33 minutes.– OlatheMar 29, 2016 at 7:37
Discovery #1: Use the "store" option for compression method, that brings it to ~33 minutes....
answered Mar 29, 2016 at 3:49
Jonesome Reinstate MonicaJonesome Reinstate Monica
5,39510 gold badges55 silver badges82 bronze badges
-
1It's not going to get much faster unless you have a really fast disk. It's almost surely IO-bound.– OlatheMar 29, 2016 at 4:47
-
@Olathe: it isn't: do the maths: 200GB/43200s is 4.7MB/s. This is hugely below the bandwidth os any modern disk.– user337366Mar 29, 2016 at 6:37
-
133 minutes is much less than 43200 seconds, which is about half a day. I get about 103 MB/s with 200GB/1980s, which seems like it might be a 7200 RPM hard drive or something like that.– OlatheMar 29, 2016 at 6:42
-
@Olathe: Oh, sorry, I was referring to the original time (there's an answer which suggest buying fancy disk/SSD for that when clearly it's not the bottleneck, and I did not read this answer properly). Yes, in this case it probably is I/O limited: especially as it has to do well over 200GB of I/O (read 200, write quite a lot). Sorry!– user337366Mar 29, 2016 at 8:54
-
Yes, this answer should really have been part of the question because I took is as such when I constructed my answer. From other comments, I believe others here also read it the same way. Mar 29, 2016 at 22:32
7zis a file compression program with optional encryption ("fastest" means the fastest file compression). As @samsmith said below in his answer to his own question, it's sped up greatly by simply turning off the compression (the time was reduced from "12 hours" to "~33 minutes"). The data rate there is about the same rate as a hard drive, so it might actually be IO-bound rather than CPU-bound.