-1

I plan to move from linux o/s a directory which is mounted on SAN(offboard) to external storage for backup. The dir size is 14TB.

I do have experience moving files in/out using rsync but never done this with large size directories/files like now.

The directories are laid down in terms on year/month/days/hours hierarchy and support raw-logs for analysis.

Problem

So with the above told, due to resource restrictions I'm not able to move complete 14TB to external/backup storage space. The backup space I have currently is max 8 TB. So is there a command or switch / parameter i can tell rsync to only copies files to the mentioned size i.e TB only.

NOTE I don't care less about the file nature its just the beyond certain size i don't want to move any data.

  • I have improved the question. Review please! – asadz Feb 24 '15 at 9:31
  • 1
    You need to decide which 6 TB of files to exclude.. and exclude them. The computer isn't going to magically know. – HopelessN00b Feb 24 '15 at 10:00
  • @HopelessN00b I don't care less about the file nature its just the beyond certain size i don't want to move any data. – asadz Feb 24 '15 at 10:23
  • 1
    Do you want to transfer only the the first 8TB of files ? – Iain Feb 24 '15 at 10:53
  • first 8TB yes under main directory /store. – asadz Feb 24 '15 at 11:23
2

You can't get rsync to stop when it has copied a certain amount of bytes by providing a limit. You can run it normally

rsync -a /src/ /dest 

and when /dest is full it will just stop with an error

 rsync: write failed on "...": No space left on device (28)

You could do something with a script

find /src -printf "%p %s\n" >filelist_size

Then use something like awk to read the filelist_size and print the file name to filelist if the sum of the filesizes is below some threshold.

You can then use the files-from=filelist argument to rsync.

It may be easier to write this in python.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.