Linux System Administration :: Resources :: Ldp LDP LGNET Issue86 Artime

An Undeletion Experience LG #86

...making Linux just a little more fun!

An Undeletion Experience
By C.E.C. Artime and J.A. Baro

We describe an attempt of recovering data unadvertedly deleted on a ext2 filesystem. Aware that the course of events was far from desirable, we are committed that our readers will not make the same mistakes and, in order to illustrate the bunch of things that one can learn during a crisis, we detail the whole story.

Prelude: the disaster spells

The arena

Hardware

Two boxes, called Fish and Bell. Fish is a Sun E250. Bell is a Pentium 3 (450 MHz). They are in the same Ethernet subnetwork.

Software

Bell runs Debian GNU/Linux 3.0r0 after a painless upgrade from 2.2r7. Kernel is Linux 2.4.18.

Fish also runs a Debian 3.0r0, port Sparc64. A recent upgrade from 2.2r7 led to a unique problem: XFree does not work for normal users (it does for root). Kernel is Linux 2.4.18-SMP.

All filesystems are ext2.

How we buggered it up

On Fish, Xfree works for root, but not for normal users.
Rationale: either a problem with permissions or user configuration files.
Solution: let us add a user and copy root's dot-files to its home directory:

root@fish# adduser judas
Enter new UNIX password:
Retype new UNIX password:
root@fish# cp --recursive /root/.[a-zA-Z]* /home/judas
root@fish# chown --recursive judas:judas /home/judas/.*

[Did you spot the difference between regexp arguments for cp and chown?]

After enough trail & error, the X problem is finally solved; it is about time to remove the user.
While trying to speed up the procedure a bit, thorough study of a terse man page reveals an appealing option:

root@fish# deluser --remove-all-files judas

Cha-ching! Enters disaster! The unfortunate option makes deluser search the entire disk looking for files owned by judas and erasing them!

Contents of /home disappeared.
Two minutes later, we unmounted its corresponding device (/dev/sda8).

Back-up

First attempt

First cares involve taking a copy of the raw bytes inside the doomed device.
There is no room in Fish, so let us make a copy onto Bell's hard drive:

root@fish# ftp bell
ftp> put '|dd if=/dev/sda8' fishbackup

In fact, this step was not so easy due to a problem with the net: transmission of big files is interrupted after a while. The file must be split. Partition /dev/sda8 is size 10142 MiB.
Resorting to the factor tool of GNU shellutils:

root@fish# factor 10142
10142: 2 11 461

This led us to an appropriate size for each chunk of device.

Secure tunnelling

Instead FTP, we used the OpenSSL suite for the next step.
First we interchange passwords in order to use batch mode:

root@fish# ssh-keygen 
artime@bell$ ssh-keygen
root@fish# scp /root/.ssh/identity.pub artime@bell:/home/artime/.ssh/authorized.keys
root@fish# scp artime@bell:~/.ssh/identity.pub ~/.ssh/authorized.keys

Let us execute this script in Fish:

#!/bin/sh
for i in $(seq 0 460)
do
  dd if=/dev/sda8 bs=$((2*11*1024)) skip=$i of=fula
  scp --batch fula artime@bell:~/fiback.$i
done

Just join the pieces in Bell:

for i in $(seq 0 460)
do cat fiback.$i >> fiback.raw
done

Equivalently we could do the same in one step by executing in Fish:

#!/bin/sh
for i in $(seq 0 460)
do
  dd if=/dev/sda8 bs=$((2*11*1024)) skip=$i | ssh artime@bell "cat >> fiback.raw"
done

Seeking lost data

We can resort to the Linux Ext2fs undeletion mini-HOWTO, by Aaron Crane; the Ext2fs undeletion of directory structures mini-HOWTO, by Tomas Ericsson; the recover program by Tom Pycke; Torsten Werner's patch for debugfs working on i386.

Lazy attempt

Recover is a tool for recovering individual files. As in our case we have suddenly lost hundreds of files, it seems this is not the best choice to begin with. The most direct way is using Werner's debugfs patch; we already have a copy of the damaged filesystem stored in a i386 box, so we can use the binary found at twerner.debian.org:

artime@bell$ ./debugfs fishback.raw
debugfs> restore
debugfs> quit
artime@bell$ mkdir mnt
artime@bell$ mount -t ext2 -o loop fishback.raw mnt
artime@bell$ ls -l mnt

We find no files there! It seems we unmounted /dev/sda8 a little too late. As T. Werner indicates on his page, his tool can only recover files that still have a name.

It pays reading

Crane's undeletion howto

This tool we pick in The Linux Documentation Project taught us how to find the inodes (more or less, the locations) of lost files and how to recover them with the dump command of debugfs. We finish with a lot of assorted files in the same directory, but they are just a small subset of the deleted ones. As mentioned before, a thorough, file by file recovery scheme as indicated in this HOWTO is unfeasible in our case.

Ericsson's directory undeletion howto

Here we learnt how to tell between deleted files from directories. We followed the indications.
The script in section 7 was in need for some further adaptation to our version of debugfs

#!/bin/sh
awk '{ print "mi <" $1 ">\n"\
             "\n\n\n\n\n\n\n"\
             "0\n"\
             "1\n"\
             "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" }'

Anyway, we fail to find the directory names and proceed to section 8. So we must look elsewhere.

Man pages

A last chance was

$ man debugfs

where we found the command rdump, unmentioned in the howtos. Applying the following script to the directory inodes located in lsdel.out of section 6 of Ext2fs Undeletion of Directory Structures mini-HOWTO:

#!/bin/sh
awk '{ print "rdump <" $1 "> folders\n" }'

most directories are recovered into the directory folders. But unnamed! So finally we have to individually assign two hundred folders to users by looking their contents! Even after that, we miss some files, notably an e-mail box. Let us try a quest by hand:

$ strings fishback.raw | grep -B1 -A99 "To: lisistrata" > lisistrata.mbox

Epilogue

Further things we would like to outline.

RTFM

And never use a new option as root for the first time.

Magnetic tapes

Fish has a magnetic tape unit. GNU mt and tar are useful tools for a backup schedule. Aaron Crane's howto includes interesting links regarding this issue.

Regexps

The use of regular expressions in shell commands is very dangerous, especially when dealing with dot-files. Perhaps the superuser root should use a shell with limited (rather than "extended") regular expression support. We are considering esh.

PDA

This article was mostly written with an Agenda VR3, a rather unique free software PDA, in a beach of Gran Canaria.

[BIO] C.E.C. Artime is a GNU fan and a free software advocator since 2000.

[BIO] J.A. Baro is a Linux user and a Perl hacker since 1996.