I'm currently copying large directory tree from an external HDD to to internal storage of my server. The performance of the copy operation is basically quite good, i.e. 35MB/s with the external HDD connected via USB2.
However, I only get this performance for about 30s and then the copy process does not get any CPU time. This means, rsync (and also cp) does not copy any data, does not update its stdout, or react on SIG-TERMs. After about 1 minute, it gets scheduled again and copies with 35MB/s again for 30s, and then nothing again and so forth.
Does anyone know what could happen here or name me any tool how I could investigate what is happening? I already increased the nice value of the rsync process to 19, but this does not change anything. During the idle time of rsync, the whole system is idle (no other IO, no other process that uses the CPU. Basically, the whole system is idle). During the active times, the rsync process has three threads that switch between 'running', 'uninterruptible sleep, and 'interruptible sleep'. During the non-active time, all threads are either have an 'uninterruptible sleep' or 'interruptible sleep' state. Because of this, I think that rsync has to wait for some long-lasting IO operation. But how could I determine what operation takes that much time?
The system is running on a 5.4.34-1-pve linux kernel and the internal storage is a LUKS encrypted ZFS file system on LVM. The LVM sits on top of a mdadm RAID 5. The command I used to start rsync is: rsync -ah --no-compress --progess
Thanks for any suggestion!