My home Linux server with all my files had a hard drive failure, but I
happily came through mostly unscathed. Your hard drive is going to
fail someday too. Backups and a rescue plan will save you, maybe these
notes will help.
My computer started acting weird; empty web page loads, ssh failing
to connect, random permission denied errors on file access, etc. I
couldn’t figure it out until I saw the dreaded error in my syslog,
hda: drive not ready for command. The moment you suspect
drive hardware failure, turn off the drive. You may have some hope still.
After a two day wait for a
new drive I unplugged all my old disks and reinstalled
Debian. Then I plugged in the old drives one by one to copy the
files off via rsync -a. I left the the known bad drive for last. Once one drive fails I
don’t trust any of the others of the same age.
I’d been faithfully running rsnapshot and so had a backup
of all my most important files on a second disk. Unfortunately I wasn’t backing up
everything, leaving me to wonder what I might have lost. rsnapshot on Debian
backs up /home, /usr/local, and /etc by
default. That seems like it’d be enough but it leaves out
/var which it turns out has important things like MySQL
databases and Apache logfiles. I’d also manually excluded my MP3
collection from the backup and I sure as
heck wasn’t going to re-rip 60 gigs of CDs.
So after getting most everything back from the backup and the other
disks I set to recover the dead drive for the last bits of files. I guessed it was a mechanical
failure so I tried
freezing
the drive first but the motherboard wouldn’t even recognize the disk.
Then I swapped hard drive logic boards; fortunately I had two identical disks.
The Frankensteined disk worked and I copied all my old files off. If the disk
had a more pernicious failure I would have tried GNU
ddrescue to make a copy of the disk image while skipping around
the bad parts of the drive.
So now I’ve got all my files back but I still have a mess. Since I wasn’t
backing up the Linux install itself I’ve lost track of what packages I
had installed before. And while I have my /etc
backup I still have to manually reconfigure all the software. So going
forward I plan to
follow jwz’s
advice and get a second disc that’s a full mirror via rsync.
What about RAID? No. RAID 1 mirroring is the only sensible thing for a
home setup and even that is bad. What if your RAID controller fails?
What if a disk fails and you don’t get notified? What if you do a
rogue rm -rf / and your RAID controller faithfully deletes
files on both disks? A versioned, time delayed backup like rsnapshot provides is way more
valuable.
Your hard drive will fail, it’s just a matter of time. Do you have a
backup and recovery plan?