There are about 130 million (129923145, precisely, by find . -name "*.*" | wc -l) pictures on server. They need to be de-duplicated, and packaged to zip compressed file from smallest to biggest for each million for later use. (say if they are 500,000 unique 8kb pictures, 600,000 unique 16kb pictures, 950,000 unique 24kb pictures, I should package 500,000 8kb pictures and 500,000 16kb pictures to the first zip file, then package 100,000 16kb pictures and 900,000 24kb pictures to the second zip file, and rest to the third), file name must be preserve, and better to keep the hierarchy information.

The server equipped with 32 GB memory, 5.5 TB hard drive space (122 GB available), CPU looks like E5-2680v4, I do not know if it is physical or virtual machine. I could ask IT mates to add memory to 512 GB, but it will takes at least one week to be approved by our Tech Director, Supply Chain guys, Budget Committee and IT department, maybe extra time to persuade them.

No extra disk space to use due to raid or something (by IT fellows). It is intranet, without internet, and I prefer not to send file in than file an application. It is Ubuntu 16.04, I am sure there are vim, python (2 and 3) and shell to use. I can only ssh in and can not sudo.

My solution to this is using du -a to make the file list, using md5sum for de-duplication, renaming all files with absolute path (replace / to __DIVIDER__), and move all file to SIZE/MD5/(fileNameWithAbsPath), than pick one for each directory. During this I met "H-tree index bug". Is there a better way (faster, more simple, etc.) to finish it, and, if possible, avoid H-tree index bug?

By the way, I am the sixth one on this duty. The former five left their job :(

  • 2
    I'm not actually sure using zip files gives you a lot of benefit here -- the compression table in regular zip format (as opposed to one of the variants supporting "solid archives") is calculated on a per-file basis, so you don't get any improvement in compression ratio over just compressing the files individually. – Charles Duffy Mar 22 at 16:39
  • Actually, we are forming our private dataset for deep learning. Packaging to zip, or rar, or tar.gz, or any other format would be fine. 1 million per package aim to be easy to ship, as moving many single little file takes too many I/O. If researchers decide to use partly data, a smallest million take least disk space and least time (when reading images). And it is also convenient for cross-validation. I am not sure if there is a better way for out purpose. – Mo Shen Mar 22 at 16:49
  • Ahh -- understanding that this is for easier network transfer, vs storage archival, makes more sense. – Charles Duffy Mar 22 at 16:55
  • BTW -- I work in security, and very, very strongly advise against using rar (particularly, against unpacking rar archives you get from others). – Charles Duffy Mar 22 at 19:15
  • 1
    When you say "8k files", are they all 8192 bytes, or is is everything between 4097 and 8192 that should go into that bucket? Is the set of sizes known and/or fairly limited? – Tollef Fog Heen Mar 22 at 19:29

So, one way you can work around the bug you ran into is splitting the size and md5 into multiple subfields. For size, you'll want to pad it out to a fixed number of digits first.

So, let's say you'd be creating a filename like:

#size / md5              / name
12345 / aabbccddeeffgghh / foo__DIVIDER__bar__DIVIDER__baz.jpg

Change it into:

# size        /md5                    /name

...and you've limited the fanout level at any given point in the tree, and thus avoided the bug in question.

To actually generate a tree of filenames in that format might look something like:

while IFS= read -r -d '' name; do
  sp=$(stat --format=%010s -- "$name") # sp as short for "size padded"


  { read -r md5 _ < <(md5sum "$name") && [[ $md5 ]]; } || continue
  while [[ $md5_left ]]; do


  if [[ -d "$final_dir" ]]; then
    # Hardlink new file to existing ones (deduplication)
    # Be sure to use an archiver that understands hardlinks (not zip)!
    existing_files=( "$final_dir"/* )
    if [[ -e "${existing_files[0]}" || -L "${existing_files[0]}" ]]; then
      ln -- "${existing_files[0]}" "$final_name"
      ln -f -- "$final_name" "$file"  # and make our input file a hardlink as well

  # if we get here, the continue was not invoked
  mkdir -p -- "${final_name%/*}"
  ln -- "$name" "$final_name"
done < <(find "$inDir" -printf '%P\0')

Of course, if you can have files larger than 9,999,999,999 bytes, you'll want to add more padding (maybe using %012s instead of %010s, and changing the calculation of size_dir appropriately).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.