46 votes
Accepted

Linux mv file with long name

for a single file try mv dir1/dir2/dir3/file.{txt,txt.old} where the X{a,b} construct expand to Xa Xb, you can have a preview using echo dir1/dir2/dir3/file.{txt,txt.old} to see if it fit your ...
Archemar's user avatar
  • 1,379
37 votes

How can I retrieve a directory (folder) from Ubuntu Server 14.04 VM without FTP, SCP, or NFS

As well as saying all the things it doesn't have, what about saying the things it does have? How big is the folder and how much free disk space is there? Assuming you have no access to the CLI at all,...
TessellatingHeckler's user avatar
35 votes

Perform rsync while following sym links

You need both -L and -K if you have symlinks on both sides, e.g. you already had your 1st rsync done and want to update the backup using rsync. -L, --copy-links transform symlink into ...
HongboZhu's user avatar
  • 451
26 votes
Accepted

Systemd drop-in fails to create PID file

The PIDFile= setting does not create a PID file. That is still up to the service itself to do, the same as it has been for the last 40 years. Rather, this option tells systemd where to find an ...
Michael Hampton's user avatar
16 votes

Copying a large directory tree locally? cp or rsync?

This thread was very useful and because there were so many options to achieve the result, I decided to benchmark few of them. I believe my results can be helpful to others have a sense of what worked ...
arjones's user avatar
  • 261
16 votes
Accepted

How to output all files/folders ordered by size, including hidden ones

How can I output all files/directories, ordered by size, including hidden ones? Use the du (disk usage) command, which is part of GNU coreutils : 1 du -hs -- * .[^.]* | sort -h The .[^.]* regular ...
Henke - Нава́льный П с м's user avatar
14 votes

How to move files between two S3 buckets with minimum cost?

Old topic, but this is for anyone investigating the same scenario. Along with the time it took me, for 20,000+ objects. Running on AWS Linux/Centos, each object being images for the most part, along ...
user1360528's user avatar
14 votes

Systemd drop-in fails to create PID file

Regrettably, systemd won't create a PID file for a non-forking service even if you specify a PIDFile= line in the service's unit file. But you may be able to cheat with an ExecStartPost= line, such as:...
Mike Gleason's user avatar
13 votes

Comparing owners and permissions of content of two folders?

Find and stat: find . -exec stat --format='%n %A %U %G' {} \; | sort > listing Run that in both directories then compare the two listing files. Saves you from the evils of Perl...
Ben's user avatar
  • 131
10 votes

How to view hidden files using Linux `find` command

It's better to use iname (case insensitive). I use this find command to search hidden files: find /path -type f -iname ".*" -ls Extracted from: http://www.sysadmit.com/2016/03/linux-ver-archivos-...
NewMailpeter's user avatar
10 votes

Extracting files from Clonezilla images

I've written a program called clonezilla-util which can mount Clonezilla archives in Windows. You can mount to a drive letter using this command (requires the Dokan driver): clonezilla-util.exe mount -...
Fidel's user avatar
  • 422
8 votes

How do I do Multihop SCP transfers between machines?

It's possible and relatively easy, even when you need to use certificates for authentication (typical in AWS environments). The command below will copy files from a remotePath on server2 directly into ...
donhector's user avatar
  • 211
8 votes

nginx split large configuration file

You can create site folders with mkdir -p /etc/nginx/sites-available /etc/nginx/sites-enabled # And then split your large your_config.conf file into smaller files into sites-available/ with: ...
rubo77's user avatar
  • 2,519
8 votes

Bash one-liner to atomically create a file if it doesn't exist

Stealing an answer from various comments and links to [SO]. It seems there is a POSIX compliant method that doesn't involve mkdir as I mentioned in my original answer below set -o noclobber # or set -...
user9517's user avatar
  • 116k
8 votes

How can I get docker build to overwrite a file?

I experienced this too, but after much testing discovered that I had a typo in one of my COPY directives' destination paths. Double-check where those files are really going -COPY nginx.conf /etc/...
ti7's user avatar
  • 225
7 votes

How to make scp copy hidden files?

To copy only hidden files, Use this command scp -rp /path_to_copy_hidden/.[!.]* user@host:/path_to_paste/ Actual game is the /.[!.]* tag that is referring to files starting with .(hidden)
Shahzaib Hayat Khan's user avatar
7 votes

Debian - how to prevent global read access for all files?

In my debian all files, even the system ones have permissions rw-r--r--, so it means that all users can view ANY file even some system configuration or database files and so on. Unless you fiddled ...
Daniel's user avatar
  • 7,059
6 votes

How do I do Multihop SCP transfers between machines?

This isn't scp (which OP requested), but I found it super simple to use rsync to copy from local to remote over a single hop with: rsync -v -e 'ssh -A -t user@jumpserver ssh -A -t user@...
texas-bronius's user avatar
6 votes

How to move files between two S3 buckets with minimum cost?

The AWS CLI provides a way to copy one bucket to another in parallel processes. Taken from https://stackoverflow.com/a/40270349/371699: The following commands will tell the AWS CLI to use 1,000 ...
timetofly's user avatar
  • 215
6 votes

How can I retrieve a directory (folder) from Ubuntu Server 14.04 VM without FTP, SCP, or NFS

The virtual machine disk is hopefully stored in a way that is backed up regularly - so you can restore the backup to a new location and then use the VM disk as a second disk in a new VM. If there are ...
Michael Shaw's user avatar
6 votes
Accepted

Can I validate a large file download piecemeal over http

On the server side, you can use dd and md5sum to checksum each chunk of the file: #!/bin/bash FILENAME="$1" FILESIZE=`stat --printf="%s" $FILENAME` CHUNKSIZE=536870912 # 512MB CHUNKNUM=0 while ! grep ...
ThoriumBR's user avatar
  • 5,332
6 votes

Linux mv file with long name

The command rename renames all matched files. Its main purpose is to rename multiple files, but if the Perl expression matches only one file, then it will rename only this single file. For details and ...
Fabian's user avatar
  • 417
5 votes
Accepted

Delete files older than X days on remote server with SCP/SFTP

This question is very old but I still wanted to add my bash only solution as I was just searching for one when I came here. The grep tar in the listing command is just for my own purpose to list only ...
dirkaholic's user avatar
5 votes
Accepted

How to maintain ssh connection for multiple scp transfers?

Nobody took the opportunity to complete answer and I didn't find any duplicate, so here we go. You can set up ~/.ssh/config, with these options: Host machine1 HostName machine1.example.org User ...
Jakuje's user avatar
  • 10.1k
5 votes

How can I retrieve a directory (folder) from Ubuntu Server 14.04 VM without FTP, SCP, or NFS

Well, there is already an accepted answer, and in addition, there is a real good answer making use of the fact it is a web server. But just to point out a line of attack that I don't think is ...
infixed's user avatar
  • 176
5 votes
Accepted

Real-time file sync between servers with hunderd-thousands of small files

If your servers are on the same LAN, then a clustered filesystem (ie: GlusterFS) or a shared storage solution (ie: via NFS) should be the better choice. If your servers are in different location, ...
shodanshok's user avatar
  • 50.7k
4 votes

How to unhide hidden NTFS folder? (option greyed out, maybe it's a system hidden folder)

Try this to remove only the hidden attribute, whilst leaving all others ad defined: $Path = 'c:\MyDemoFile.txt' #use -force switch with get-item so we find the file even if it's hidden $Item = (get-...
JohnLBevan's user avatar
  • 1,312
4 votes

Copying a large directory tree locally? cp or rsync?

There are some speed-ups which can be applied to rsync: Avoid -z/--compress: compression will only load up the CPU as the transfer isn't over a network but over RAM. --append-verify: resume an ...
Tom Hale's user avatar
  • 1,165
4 votes

Copying a large directory tree locally? cp or rsync?

You definitely want to give rclone a try. This thing is crazy fast : sudo rclone sync /usr /home/fred/temp -P -L --transfers 64 Transferred: 17.929G / 17.929 GBytes, 100%, 165.692 MBytes/s, ...
Frédéric N.'s user avatar

Only top scored, non community-wiki answers of a minimum length are eligible