I have dump directory that is updated regularly.
I wish to delete all files that are older than one week.
Bash is preferred though other solution are also welcomed.
I have dump directory that is updated regularly.
I wish to delete all files that are older than one week.
Bash is preferred though other solution are also welcomed.
The trick is the --full-time
flag, which can be given to an ls
command, which gives back a file list with very easily scriptable date fields. We can easily sort it by the date.
$ ls -l --full-time|sort -k +7
-rwxr-xr-x 1 cica cica 16536 2014-07-10 10:47:32.448349200 +0200 epl-v10.html
-rwxr-xr-x 1 cica cica 9013 2014-07-10 10:47:32.495149500 +0200 notice.html
drwxr-xr-x+ 1 cica cica 0 2014-07-07 14:12:11.502336700 +0200 readme
drwxr-xr-x+ 1 cica cica 0 2014-07-31 14:38:20.800181400 +0200 p2
drwxr-xr-x+ 1 cica cica 0 2014-07-31 15:15:06.506730000 +0200 features
drwxr-xr-x+ 1 cica cica 0 2014-07-31 15:15:06.680747400 +0200 plugins
-rw-r--r-- 1 cica cica 368634 2014-07-31 15:15:06.826762000 +0200 artifacts.xml
-rw-r--r-- 1 cica cica 329 2014-07-31 15:15:12.816360900 +0200 eclipse.ini
-rwxr-xr-x 1 cica cica 7929 2014-07-14 16:01:58.698363500 +0200 system_catalog.xml
drwxr-xr-x+ 1 cica cica 0 2014-07-31 17:41:59.205940000 +0200 configuration
-rwxr-xr-x 1 cica cica 312320 2014-06-01 20:12:16.000000000 +0200 eclipse.exe
-rwxr-xr-x 1 cica cica 17920 2014-06-01 20:12:16.000000000 +0200 eclipsec.exe
First, we sort it by the full date!
The trick is, that after the each days last file, the date field will replay after the previous line. This can be handled by awk very easily.
Second, we get it further to a simple awk script: awk '{if ($6 == EX) print $9; EX=$6}'
Finally, we are using xargs
to let delete every file with an rm
command.
The full command is:
ls -l --full-time|sort -k +6|awk '{if ($6 == EX) print $9; EX=$6}'|xargs -P 1 -n 1 echo rm -vf
This command is what you need to call periodically, ideally from a cron. Ideally, you can give into a crontab -e
, you call this every day at 2:37 :
37 2 * * * ls -l --full-time|sort -k +6|awk '{if ($6 == EX) print $9; EX=$6}'|xargs -P 1 -n 1 echo rm -vf
Of course you could put this in a script and call only the script from cron.
I was looking for a similar thing but couldn't find anything, wrote my own script which does this. I'm fairly inexperienced with bash so this could probably be done a lot cleaner and faster but this worked for me. I realize that this question is quite old but still wanted to answer it.
What this specific script does:
Optimizations that could be done is modifying the hard delete to just use find with the delete and mtime flags, but I wanted to keep it all together.
#!/bin/bash
full_delete=$((60 * 60 * 24 * 30)) # 30 days
partial_delete=$((60 * 60 * 24 * 5)) # 5 days
for filename in ./data/*; do
# Check if file still exists, partial delete might've nuked it
if [ -f "$filename" ]; then
# Get epoch diff between now and when it was last modified
since_modified_epoch=$(( $(date +%s) - $(stat -L --format %Y "$filename") ))
# File is past our hard threshold, just remove
if [ $since_modified_epoch -gt $full_delete ]; then
rm "$filename"
elif [ $since_modified_epoch -gt $partial_delete ]; then
# Y-m-D file was modified
modified_date=$(date +%Y-%m-%d -r "$filename")
# Y-m-D file was modified + 1 day
modified_date_plus=$(date +%Y-%m-%d -d "$modified_date +1 days")
# Get all files where modified date is between previous 2 dates
# 'head -n -1' ignores the newest file
partial_files=$( find ./backups/ -type f -newermt "$modified_date" -not -newermt "$modified_date_plus" -printf "%T+§%p\\n" | sort | head -n -1 )
for partial_file in $partial_files; do
# Extract just filename from the line
partial_to_delete=$( echo "$partial_file" | cut -d'§' -f 2 )
rm "$partial_to_delete"
done;
fi
fi
done
Some code you could use to generate a test case. This will generate 40 days worth of 4-hour interval files.
#!/bin/bash
rm -rf ./data
mkdir -p data
for i in $(seq 1 $(( 40 * (24 / 4) )) ); do
amount=$(( i * 4 * 60 ))
touch -d "$amount minutes ago" "./data/data_$i";
done
The trick is the --full-time
flag, which can be given to an ls
command, which gives bacvk a file list with very good scriptable date fields:
$ ls -l --full-time|sort -k +7
-rwxr-xr-x 1 cica cica 16536 2014-07-10 10:47:32.448349200 +0200 epl-v10.html
-rwxr-xr-x 1 cica cica 9013 2014-07-10 10:47:32.495149500 +0200 notice.html
drwxr-xr-x+ 1 cica cica 0 2014-07-07 14:12:11.502336700 +0200 readme
drwxr-xr-x+ 1 cica cica 0 2014-07-31 14:38:20.800181400 +0200 p2
drwxr-xr-x+ 1 cica cica 0 2014-07-31 15:15:06.506730000 +0200 features
drwxr-xr-x+ 1 cica cica 0 2014-07-31 15:15:06.680747400 +0200 plugins
-rw-r--r-- 1 cica cica 368634 2014-07-31 15:15:06.826762000 +0200 artifacts.xml
-rw-r--r-- 1 cica cica 329 2014-07-31 15:15:12.816360900 +0200 eclipse.ini
-rwxr-xr-x 1 cica cica 7929 2014-07-14 16:01:58.698363500 +0200 system_catalog.xml
drwxr-xr-x+ 1 cica cica 0 2014-07-31 17:41:59.205940000 +0200 configuration
-rwxr-xr-x 1 cica cica 312320 2014-06-01 20:12:16.000000000 +0200 eclipse.exe
-rwxr-xr-x 1 cica cica 17920 2014-06-01 20:12:16.000000000 +0200 eclipsec.exe
First, we sort it by the full date!
The trick is, that after the each days last file, the date field will replay after the previous line. This can be handled by awk very easily.
Second, we get it further to a simple awk script: awk '{if ($6 == EX) print $9; EX=$6}'
Finally, we are using xargs
to let delete every file with an rm
command.
The full command is:
ls -l --full-time|sort -k +6|awk '{if ($6 == EX) print $9; EX=$6}'|xargs -P 1 -n 1 echo rm -vf
you can also use find command. We’ll use this in order to figure out what files are older than a certain number of days, and then use the rm command to delete them.
find <path of file> -typf f -mtime +0 -exec rm {} \;
it will delete files which are dated yesterday.