There are about 130 million (129923145, precisely, by
find . -name "*.*" | wc -l) pictures on server. They need to be de-duplicated, and packaged to zip compressed file from smallest to biggest for each million for later use. (say if they are 500,000 unique 8kb pictures, 600,000 unique 16kb pictures, 950,000 unique 24kb pictures, I should package 500,000 8kb pictures and 500,000 16kb pictures to the first zip file, then package 100,000 16kb pictures and 900,000 24kb pictures to the second zip file, and rest to the third), file name must be preserve, and better to keep the hierarchy information.
The server equipped with 32 GB memory, 5.5 TB hard drive space (122 GB available), CPU looks like E5-2680v4, I do not know if it is physical or virtual machine. I could ask IT mates to add memory to 512 GB, but it will takes at least one week to be approved by our Tech Director, Supply Chain guys, Budget Committee and IT department, maybe extra time to persuade them.
No extra disk space to use due to raid or something (by IT fellows). It is intranet, without internet, and I prefer not to send file in than file an application. It is Ubuntu 16.04, I am sure there are vim, python (2 and 3) and shell to use. I can only ssh in and can not
My solution to this is using
du -a to make the file list, using
md5sum for de-duplication, renaming all files with absolute path (replace
__DIVIDER__), and move all file to
SIZE/MD5/(fileNameWithAbsPath), than pick one for each directory. During this I met "H-tree index bug". Is there a better way (faster, more simple, etc.) to finish it, and, if possible, avoid H-tree index bug?
By the way, I am the sixth one on this duty. The former five left their job :(