-1

I have dump directory that is updated regularly.

I wish to delete all files that are older than one week.

Bash is preferred though other solution are also welcomed.

3
  • 2
    I would do this by making sure that the files have a consistent naming convention and then use find's ! operator to delete all bar the files I wanted to keep.
    – user9517
    Aug 1, 2014 at 12:34
  • @Iain, Will do that unless its possible to achieve simply by relaying on the creation time. Aug 1, 2014 at 12:46
  • At the end I used name conventions as suggested by @lain. I would prefer something more 'less responsible' but for the sake of simplicity that will be enough. Aug 4, 2014 at 7:59

4 Answers 4

5

The trick is the --full-time flag, which can be given to an ls command, which gives back a file list with very easily scriptable date fields. We can easily sort it by the date.

$ ls -l --full-time|sort -k +7
-rwxr-xr-x  1 cica cica  16536 2014-07-10 10:47:32.448349200 +0200 epl-v10.html
-rwxr-xr-x  1 cica cica   9013 2014-07-10 10:47:32.495149500 +0200 notice.html
drwxr-xr-x+ 1 cica cica      0 2014-07-07 14:12:11.502336700 +0200 readme
drwxr-xr-x+ 1 cica cica      0 2014-07-31 14:38:20.800181400 +0200 p2
drwxr-xr-x+ 1 cica cica      0 2014-07-31 15:15:06.506730000 +0200 features
drwxr-xr-x+ 1 cica cica      0 2014-07-31 15:15:06.680747400 +0200 plugins
-rw-r--r--  1 cica cica 368634 2014-07-31 15:15:06.826762000 +0200 artifacts.xml
-rw-r--r--  1 cica cica    329 2014-07-31 15:15:12.816360900 +0200 eclipse.ini
-rwxr-xr-x  1 cica cica   7929 2014-07-14 16:01:58.698363500 +0200 system_catalog.xml
drwxr-xr-x+ 1 cica cica      0 2014-07-31 17:41:59.205940000 +0200 configuration
-rwxr-xr-x  1 cica cica 312320 2014-06-01 20:12:16.000000000 +0200 eclipse.exe
-rwxr-xr-x  1 cica cica  17920 2014-06-01 20:12:16.000000000 +0200 eclipsec.exe

First, we sort it by the full date!

The trick is, that after the each days last file, the date field will replay after the previous line. This can be handled by awk very easily.

Second, we get it further to a simple awk script: awk '{if ($6 == EX) print $9; EX=$6}'

Finally, we are using xargs to let delete every file with an rm command.

The full command is:

ls -l --full-time|sort -k +6|awk '{if ($6 == EX) print $9; EX=$6}'|xargs -P 1 -n 1 echo rm -vf

This command is what you need to call periodically, ideally from a cron. Ideally, you can give into a crontab -e, you call this every day at 2:37 :

37 2 * * *     ls -l --full-time|sort -k +6|awk '{if ($6 == EX) print $9; EX=$6}'|xargs -P 1 -n 1 echo rm -vf

Of course you could put this in a script and call only the script from cron.

6
  • But this doesn't do what the requestor asked!
    – MadHatter
    Aug 1, 2014 at 12:32
  • @MadHatter Yes, it is true. And now?
    – peterh
    Aug 1, 2014 at 12:35
  • 2
    Umm, still not. After a week he wants to delete all except for each day's youngest (newest) file.
    – MadHatter
    Aug 1, 2014 at 12:35
  • 4
    As an aside linux find has a -delete option which is quite handy
    – user9517
    Aug 1, 2014 at 12:38
  • I haven't voted.
    – user9517
    Aug 1, 2014 at 18:30
1

I was looking for a similar thing but couldn't find anything, wrote my own script which does this. I'm fairly inexperienced with bash so this could probably be done a lot cleaner and faster but this worked for me. I realize that this question is quite old but still wanted to answer it.

What this specific script does:

  • Permanently removes files older than 'full_delete'.
  • Keep most recent of each day for files older than 'partial_delete'
  • Keep all other files

Optimizations that could be done is modifying the hard delete to just use find with the delete and mtime flags, but I wanted to keep it all together.

#!/bin/bash

full_delete=$((60 * 60 * 24 * 30)) # 30 days
partial_delete=$((60 * 60 * 24 * 5)) # 5 days

for filename in ./data/*; do
  # Check if file still exists, partial delete might've nuked it
  if [ -f "$filename" ]; then

    # Get epoch diff between now and when it was last modified
    since_modified_epoch=$(( $(date +%s) - $(stat -L --format %Y "$filename") ))

    # File is past our hard threshold, just remove
    if [ $since_modified_epoch -gt $full_delete ]; then
      rm "$filename"

    elif [ $since_modified_epoch -gt $partial_delete ]; then
      # Y-m-D file was modified
      modified_date=$(date +%Y-%m-%d -r "$filename")
      # Y-m-D file was modified + 1 day
      modified_date_plus=$(date +%Y-%m-%d -d "$modified_date +1 days")
      # Get all files where modified date is between previous 2 dates
      # 'head -n -1' ignores the newest file
      partial_files=$( find ./backups/ -type f -newermt "$modified_date" -not -newermt "$modified_date_plus" -printf "%T+§%p\\n" | sort | head -n -1 )

      for partial_file in $partial_files; do
        # Extract just filename from the line
        partial_to_delete=$( echo "$partial_file" | cut -d'§' -f 2 )
        rm "$partial_to_delete"
      done;
    fi
  fi
done

Some code you could use to generate a test case. This will generate 40 days worth of 4-hour interval files.

#!/bin/bash
rm -rf ./data
mkdir -p data

for i in $(seq 1 $(( 40 * (24 / 4) )) ); do
  amount=$(( i * 4 * 60 ))
  touch -d "$amount minutes ago" "./data/data_$i";
done
0

The trick is the --full-time flag, which can be given to an ls command, which gives bacvk a file list with very good scriptable date fields:

$ ls -l --full-time|sort -k +7
-rwxr-xr-x  1 cica cica  16536 2014-07-10 10:47:32.448349200 +0200 epl-v10.html
-rwxr-xr-x  1 cica cica   9013 2014-07-10 10:47:32.495149500 +0200 notice.html
drwxr-xr-x+ 1 cica cica      0 2014-07-07 14:12:11.502336700 +0200 readme
drwxr-xr-x+ 1 cica cica      0 2014-07-31 14:38:20.800181400 +0200 p2
drwxr-xr-x+ 1 cica cica      0 2014-07-31 15:15:06.506730000 +0200 features
drwxr-xr-x+ 1 cica cica      0 2014-07-31 15:15:06.680747400 +0200 plugins
-rw-r--r--  1 cica cica 368634 2014-07-31 15:15:06.826762000 +0200 artifacts.xml
-rw-r--r--  1 cica cica    329 2014-07-31 15:15:12.816360900 +0200 eclipse.ini
-rwxr-xr-x  1 cica cica   7929 2014-07-14 16:01:58.698363500 +0200 system_catalog.xml
drwxr-xr-x+ 1 cica cica      0 2014-07-31 17:41:59.205940000 +0200 configuration
-rwxr-xr-x  1 cica cica 312320 2014-06-01 20:12:16.000000000 +0200 eclipse.exe
-rwxr-xr-x  1 cica cica  17920 2014-06-01 20:12:16.000000000 +0200 eclipsec.exe

First, we sort it by the full date!

The trick is, that after the each days last file, the date field will replay after the previous line. This can be handled by awk very easily.

Second, we get it further to a simple awk script: awk '{if ($6 == EX) print $9; EX=$6}'

Finally, we are using xargs to let delete every file with an rm command.

The full command is:

ls -l --full-time|sort -k +6|awk '{if ($6 == EX) print $9; EX=$6}'|xargs -P 1 -n 1 echo rm -vf
0

you can also use find command. We’ll use this in order to figure out what files are older than a certain number of days, and then use the rm command to delete them.

find <path of file> -typf f -mtime +0 -exec rm {} \;

it will delete files which are dated yesterday.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .