I encountered this error running rsync locally (from a python script), not by dir, but looping through dirs and subdirs via os.walk to get to the folders that have files, and then rsync'ing each file with multiprocessing.pool with starmap. I searched for an answer for this scenario for a long time, and finally I found this link that presented the solution;
Through os.walk, I grabbed the root dir, and if there were files associated with that root dir, I then call rsync with the -aR flags;
import subprocess
import os
import sys
import multiprocessing
from util_subprocess import exec_subprocess
ORIG_SRC_ROOT = '/home/username/workspaces/data/prod'
# IMPORTANT (slash, period) - we need to tell rsync
# that we want it to create subdirs relatively after the period
# in the DEST_ROOT dir
REL_SRC_ROOT = ORIG_SRC_ROOT + '/.'
# rsync will create the directory structure for you below the DEST_ROOT
DEST_ROOT = '/home/username/workspaces/data/prod_bak'
CPU_COUNT = multiprocessing.cpu_count()
if(not CPU_COUNT):
CPU_COUNT = 1
def sync(src, dest):
'''
Make the call the subprocess.run to execute rsync.
NOTE:
The flags are key here:
-- '-a' (-a, --archive archive mode; equals -rlptgoD (no -H,-A,-X))
is pretty standard and covers most everything we need.
-- '-R'(-R, --relative use relative path names),
makes sure rsync creates the subdirs on the destination folder.
'''
flags: str = r"-aR"
program_name: str = r"rsync"
exec_list = [program_name, flags , src, dest]
return exec_subprocess(exec_list)
def get_sync_params():
'''Get a list of tuples, each tuple containing the src_file path, and dest_dir path.'''
sync_params = []
for root, dirs, files in os.walk(ORIG_SRC_ROOT, topdown=False):
if(files):
for file in files:
src_file_temp = os.path.join(root, file)
# replace the orig root dir with the relative one from above
src_file = src_file_temp.replace(ORIG_SRC_ROOT, REL_SRC_ROOT)
dest_dir = DEST_ROOT
# place them into a tuple
sync_tuple = (src_file, dest_dir)
# add the tuple to the list
sync_params.append(sync_tuple)
return sync_params
if __name__ == "__main__":
'''Create a pool for multiprocessing and get these done in parallel.'''
# call the get_sync_params above
sync_params = get_sync_params()
# Create a pool of specific number of CPUs
print("Create {} processes in our pool.".format(CPU_COUNT))
pool = multiprocessing.Pool(processes=CPU_COUNT)
'''
Start each task within the pool -
call the 'sync' function above
(which calls a separate wrapper function
to the subprocess.run call (exec_subprocess)).
The return value is a list of tuples
and each tuple contains the output (stdout OR stderr)
and the return_code results for each pair of src & dest input.
'''
results = pool.starmap(sync, sync_params)
So essentially we create a list of tuples where each tuple is the src_file path and the dest_dir path, handled by multiprocessing.pool.starmap(function_name, tuple_list), which in turn uses a process in the pool for each tuple which is sent to rsycn over subprocess.run. A tuple in the list might look like;
[('/home/username/workspaces/data/prod/./math/alg_1/alg_1_test.txt',
'/home/username/workspaces/data/prod_bak/'), ('...', '...'), ]
The first string is the src_file path, the second is the dest_dir path. Notice the '.' in the src_file path (telling rsync we need to create subdirs after this in the destination), and the trailing '/' in the dest_dir path, telling rsync to put those new subdirs under this destination root dir.
