41

Sometimes people delete files they shouldn't, a long-running process still has the file open, and recovering the data by catting /proc/<pid>/fd/N just isn't awesome enough. Awesome enough would be if you could "undo" the delete by running some magic option to ln that would let you re-link to the inode number (recovered through lsof).

I can't find any Linux tools to do this, least with cursory Googling.

What do you got, serverfault?

EDIT1: The reason catting the file from /proc/<pid>/fd/N isn't awesome enough is because the process which still has the file open is still writing to it. A delete removes the reference to the inode from the filesystem namespace. What I want is a way of re-creating the reference.

EDIT2: 'debugfs ln' works but the risk is too high since it frobs raw filesystem data. The recovered file is also crazy inconsistent. The link count is zero and I can't add links to it. I'm worse off this way since I can just use /proc/<pid>/fd/N to access the data without corrupting my fs.

9 Answers 9

20

Awesome enough would be if you could "undo" the delete by running some magic option to ln that would let you re-link to the inode number (recovered through lsof).

This awesomeness was introduced to ln in v8.0 (GNU/coreutils) with the -L|--logical option which causes ln to dereference a /proc/<pid>/fd/<handle> first. So a simple

ln -L /proc/<pid>/fd/<handle> /path/to/deleted/file

is enough to relink a deleted file.

6
  • 13
    This doesn't work; if the file is deleted it will fail.
    – Random832
    Commented Jun 20, 2011 at 22:47
  • 4
    This sort of functionality (See this question and answer) was specifically rejected as a security risk [to a hypothetical security scheme revolving around a privileged process giving your process a read-only file handle to a file you own but do not otherwise have access to]; I tried it (though with a small C program to directly use the relevant system call) and it did not work.
    – Random832
    Commented Aug 24, 2011 at 19:28
  • 11
    I, of course, did test this before posting my solution and at that time it actually worked for me. What I wasn't aware of is that it only worked on tmpfs filesystems but not on e.g. ext3. Furthermore this feature got completely disabled in 2.6.39, see the commit. So therefore this solution won't work with kernel 2.6.39 or newer anymore and in earlier versions it depends on the filesystem.
    – tnimeu
    Commented Jan 26, 2012 at 14:04
  • 9
    @tnimeu ln -L does not work for me. I have a deleted file and I tried to relink it to the original path. ln gives me a ln: failed to create hard link /my/path/file.pdf => /proc/19674/fd/16: No such file or directory. But I can e.g. successfully cat /proc/19674/fd/16 Commented Aug 21, 2013 at 6:46
  • 5
    Don't bother with ln -L, it's been disabled in the Linux kernel since 2011 due to security concerns.
    – rustyx
    Commented Apr 29, 2020 at 21:36
14

It sounds like you already understand a lot, so I won't go into excess detail. There's several methods to find the inode and you can usually cat and redirect STDOUT. You can use debugfs. Run this command within:

ln <$INODE> FILENAME

Make sure you have backups of the filesystem. You'll probably need to run a fsck afterwards. I tested this successfully with an inode still being written to and it does work to create a new hard link to a dereferenced inode.

If the file is unlinked with an unopen file in ext3, the data is lost. I'm not sure how consistently true this is but most of my data recovery experience is with ext2. From the ext3 FAQ:

Q: How can I recover (undelete) deleted files from my ext3 partition? Actually, you can't! This is what one of the developers, Andreas Dilger, said about it:

In order to ensure that ext3 can safely resume an unlink after a crash, it actually zeros out the block pointers in the inode, whereas ext2 just marks these blocks as unused in the block bitmaps and marks the inode as "deleted" and leaves the block pointers alone.

Your only hope is to "grep" for parts of your files that have been deleted and hope for the best.

There's also relevant information in this question:

I overwrote a large file with a blank one on a linux server. Can I recover the existing file?

6
  • Comment hopefully deleted as it was not revelent.
    – mdpc
    Commented Aug 10, 2010 at 16:39
  • 2
    In the case of a deleted but still open file I don't think it would zero out the pointers in the inode. Also, instead of using "ln" in debugfs I'd use "undel" so that the inode reference counts will correctly get updated. Commented Aug 10, 2010 at 19:13
  • I didn't mean to allude as such, embobo. It doesn't, I tested the performance. I've clarified my language.
    – Warner
    Commented Aug 10, 2010 at 19:21
  • Clever, but corrupts my filesystem. :)
    – mbac32768
    Commented Aug 11, 2010 at 17:32
  • It's the only solution for the scenario as you describe it. Manipulating a filesystem on a low level that is mounted rw and actively being written to is likely to cause corruption in almost all scenarios.
    – Warner
    Commented Aug 11, 2010 at 17:44
9

the debugfs way as you saw doesn't really work and at best your file will be deleted automatically (due to the journal) after reboot and at worst you can trash your filesystem resulting to "reboot cycle of death". The Right Solution (TM) is to perform the undelete in the VFS level (which also has the added benefit of working with practically all current Linux filesystems). The system call way (flink) has been shot down every time it appeared in LKML so the best way is through a module + ioctl.

A project that implements this approach and has reasonably small and clean code is fdlink (https://github.com/pkt/fdlink.git for a version tested with ubuntu maverick's kernel). With it, after you insert the module (sudo insmod flink_dev.ko) you can just do "./flinkapp /proc//fd/X /my/link/path" and it will do exactly what you want.

You can also use a forward-ported version of vfs-undelete.sourceforge.net that also works (and can also automatically relink to the original name), but fdlink's code is simpler and it works just as well, so it is my preference.

8

Ran into the same problem today. The best I could come up with is to run

% tail -n +0 -f /proc/<pid>/fd/<fd> >/path/to/write/restored_file

in a tmux/screen session until the process ends.

3
  • 2
    Linking to the original files, as in the accepted answer, should work.
    – Chris S
    Commented Dec 6, 2012 at 16:15
  • 1
    There's no accepted answer for this question, which one are you referring to? Commented Feb 4, 2015 at 20:57
  • 5
    Shouldn't this need a redirection (>) to the deleted file?
    – noe
    Commented Sep 13, 2018 at 13:20
5

I don't know how to do exactly what you want, but what I would do is:

  • Open the file RO from another process
  • Wait for the original process to exit
  • Copy the data from your open FD to a file

Not ideal, obviously, but possible. The other option is to play around with debugfs (using the link command), but that's kind of scary on a production machine!

3
  • The debugfs link command doesn't support this use case at all.
    – mbac32768
    Commented Aug 10, 2010 at 16:53
  • tldp.org/HOWTO/Ext2fs-Undeletion-11.html suggests that it does. I haven't tried it, but that seems reasonable.
    – Bill Weiss
    Commented Aug 10, 2010 at 18:18
  • link didn't work in my testing but ln did.
    – Warner
    Commented Aug 10, 2010 at 19:16
3

The quick solution that worked for me, without intimidating tools:

1) find the process+fd by looking directly in /proc:

ls -al /proc/*/fd/* 2>/dev/null | grep {filename}

2) Then a similar technique to @nickray's, with pv thrown in:

tail -c +0 -f /proc/{procnum}/fd/{fdnum} | pv -s {expectedsize} > {recovery_filename}

You may need to Ctrl-C when done (ls /proc/{procnum}/fd/{fdnum} will tell you that the file no longer exists)), but if you know the exact size in bytes, you can use pv -S to make it exit when the count is reached.

2

Interesting question. An interviewer asked the same question to me in a job interview. What I told him was that there was not a easy way to do this and in general was not worth the time and effort involved. I did ask him what he thought the solution to this issue was ....

  1. Use lsof to find the inode number on the disk for the process as it will still appear even if the file has been deleted...the key is that it is still open.
  2. Extract the information from the filesystem based on this via a filesystem debugger.
1
  • 1
    I can extract the data just fine from /proc/<pid>/fd/N but that's not what I'm trying to do.
    – mbac32768
    Commented Aug 10, 2010 at 16:51
2

Use Sleuthkit icat.

sudo icat /dev/rdisk1s2 5484287 > accidentally_deleted_open_file
1
  • 2
    This works by bypassing the operating system's filesystem functionality and parsing the disk bytes directly.
    – Flimm
    Commented Jul 3, 2014 at 12:24
0

From open (2) manual:

...
       O_TMPFILE (since Linux 3.11)
              Create  an  unnamed  temporary  regular file.  The pathname argument specifies a directory; an unnamed inode will be created in that direc‐
              tory's filesystem.  Anything written to the resulting file will be lost when the last file descriptor is closed, unless the file is given a
              name.

              O_TMPFILE must be specified with one of O_RDWR or O_WRONLY and, optionally, O_EXCL.  If O_EXCL is not specified, then linkat(2) can be used
              to link the temporary file into the filesystem, making it permanent, using code like the following:

                  char path[PATH_MAX];
                  fd = open("/path/to/dir", O_TMPFILE | O_RDWR,
                                          S_IRUSR | S_IWUSR);

                  /* File I/O on 'fd'... */

                  linkat(fd, "", AT_FDCWD, "/path/for/file", AT_EMPTY_PATH);

                  /* If the caller doesn't have the CAP_DAC_READ_SEARCH
                     capability (needed to use AT_EMPTY_PATH with linkat(2)),
                     and there is a proc(5) filesystem mounted, then the
                     linkat(2) call above can be replaced with:

                  snprintf(path, PATH_MAX,  "/proc/self/fd/%d", fd);
                  linkat(AT_FDCWD, path, AT_FDCWD, "/path/for/file",
                                          AT_SYMLINK_FOLLOW);
                  */

              In this case, the open() mode argument determines the file permission mode, as with O_CREAT.

              Specifying O_EXCL in conjunction with O_TMPFILE prevents a temporary file from being linked into the filesystem in the above manner.  (Note
              that the meaning of O_EXCL in this case is different from the meaning of O_EXCL otherwise.)

              There are two main use cases for O_TMPFILE:

              •  Improved  tmpfile(3)  functionality: race-free creation of temporary files that (1) are automatically deleted when closed; (2) can never
                 be reached via any pathname; (3) are not subject to symlink attacks; and (4) do not require the caller to devise unique names.

              •  Creating a file that is initially invisible, which is then populated with data and adjusted to have  appropriate  filesystem  attributes
                 (fchown(2),  fchmod(2), fsetxattr(2), etc.)  before being atomically linked into the filesystem in a fully formed state (using linkat(2)
                 as described above).

              O_TMPFILE requires support by the underlying filesystem; only a subset of Linux filesystems provide that support.  In the initial implemen‐
              tation,  support  was  provided in the ext2, ext3, ext4, UDF, Minix, and tmpfs filesystems.  Support for other filesystems has subsequently
              been added as follows: XFS (Linux 3.15); Btrfs (Linux 3.16); F2FS (Linux 3.16); and ubifs (Linux 4.9)
...

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .