Undeleting an open file by inode

Submitted by dag on Mon, 2008/08/04 - 17:49

At a customer today I was confronted with a situation where VMware ESX processes had log-files open that were already deleted. This can happen when logrotate was incorrectly configured, or when operational staff removed big files to clean up diskspace quickly.

However when the file is still open, you can remove the file-entry (link) to the inode, but the diskspace will not become available until all kernel references to the inode are gone (and a process having the file open counts as a reference too).

Killing the VMware processes that had the file open was not an option since we just wanted to truncate the file without impacting the guests.

There are 2 possibilities to truncate the file. The first one is very easy. First we have to find the entry in the /proc filesystem:

[root@system root]# ls -l /proc/2334/fd/ | grep delete
lrwx------ 1 root root 64 Aug 4 18:34 3 -> /path/to/vmware.log (deleted)

Now we know the file-descriptor, we can do and truncate the (unreal) symlink to the inode:

[root@system root]# echo -n >/proc/2334/fd/3

Now this procedure is safe (given that the process did not do anything with the file-descriptors in between), but it will not help with recovering the file itself (in case you wanted that).

So the longer procedure involves debugfs. By using lsof one can find the inode of the open file:

[root@system root]# lsof -p 2334 | grep delete
vmware-vm 2334 root 3u REG 8,5 43003904 46087 /path/to/vmware.log (deleted)

This gives us the original location (/path/to/vmware.log) and the inode (46087) of the deleted file. Now we have to find out on what filesystem/device this inode is from. The easiest is:

[root@system root]# df -h /path/to/
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 1.7G 1.7G 0 100% /path

Beware: the below commands are not for the faint of heart. If you do something wrong you might trash your filesystem, crash your system or inadvertently kill a dozen puppies.

So we now know the device is /dev/sda5 and we can start creating a file-entry to the inode on the correct filesystem by doing:

[root@system root]# debugfs -w /dev/sda5 -R 'link <46087> /path/to/vmware.log'

Now the file entry is back, albeit it has no links referenced to it, so it is still considered to be deleted by the filesystem if the file is eventually closed.

[root@system root]# ls -l /path/to/vmware.log
-rw-r--r-- 0 vmware vmware 43003904 Apr 18 14:48 /path/to/vmware.log

Now you can access the original content, or you can truncate the file to free up diskspace.

[root@system root]# echo -n >/path/to/vmware.log

If you happen to know how one can also restore the number of links of that inode so that the file is in fact restored, please let me know. I tried using modify_inode <46087> but that did not update the inode information when leaving debugfs.

fdlink will do that for you

About a year ago I spent a weekend creating a kernel module and a user-space program to achieve just that - given a path to the file descriptor under /proc/nnn/fd/mmm and a name to link to it on the same file system, it'll create a directory entry for that inode.

It worked for me when tested a few times (tested it on loop-mounted filesystems so could test with a few types of filesystems and not care about disasters). I never completed it into a finished application but it's pretty useful as it is.

The nice thing with it is that you should be able to download, compile and install the module without taking the system down so you won't loose the process which keeps the file "alive".

Link to the SourceForge project (also in the "home page" above): https://sourceforge.net/projects/fdlink/

Make the link number correct

I suggest to create link by "ln" system command after creating the link by debugfs. Then you can delete one of the new links by debugfs "unlink". The link count should be ok then as we balanced calls to "ln" and "unlink" debugfs commands.

doesn't work

I just tried that, but ln(1) doesn't like it:

ln: failed to create hard link `file' => `file2': No such file or directory

Opening Inode Files

My Linux machine crashed. Finally when I could recover some data (using EaseUS), I got few RAW Files and a number of folders showing names like Inode 00aFFe and associated files. Can I recover some data from these