After this server got slashdotted, it got a bit confused and broken. Some daemons stopped and it all went a bit weird. Because I still retain some traces of a Microsoft ancestry, and I'm not fully au fait with the best way to do stuff to a Unix box without rebooting it (what is a runlevel, anyway?), I decided to reboot it, so it would come back up OK and carry on serving pages. However, the reboot command was ignored. I thought, hm, that's weird, and forgot about it. A couple of weeks passed, and I needed to add some mysql stuff. Couldn't connect with the mysql client. Couldn't restart the daemon. Decided to reboot again. Again, the command failed. So I forced a reboot. And the box didn't come back up. Kernel panic on reboot; bread failed. (Ha! thought I, the bread has failed, eh? Bad dough or something. Of course, it's b-read -- binary read, if you will.)

So I panicked, much like the kernel. Imagined the worst. Cried about how the drive was probably scragged. Was rudely awakened as to the lack of a backup strategy. Oops.

You know the feeling you get when you know you've done something wrong and there's nothing you can do about it other than live with the consequences? Yeah. Just like that. Oof.

So my hosting guy, a true friend in adversity, brought angel (this box) round to me at work. I took it home and tried booting it. Sure enough, kernel panic; bread failed. Swore a lot. So I took the hard drive out of the box and put it in giles, my home machine, and mounted the partitions. Everything was there.

You know the feeling of sweaty and unwarranted relief when you teetered on the edge of a yawning chasm of fuck-up-ness and are retrieved by nothing more than blind luck? Yeah. Just like that. Oof.

So I took backups of all of it and stored them in two places, in case the drive was still scragged. And then I spent a day playing with lilo, trying to work out how to tell it "write a boot block to the second drive, which will boot to the kernel on the second drive, but that second drive will be the first drive when it boots." And I failed. Abjectly. To the point where I thought about maybe installing grub or something instead, because it must be lilo's fault. Swore more. Cried a bit. Asked for help. Got nowhere. The drive just refused to boot OK, but all the data was fine. I was most puzzled.

Then it dawned on me what the problem was. You see, my lilo.conf has the root partition as /dev/hda10. Now, that's wrong. It should have been /dev/hda1, which you might expect. So, when I saw that, I glossed over it; after all, that must have just been weirdness. But it wasn't; it was the whole problem. You see, all my playing around with the drive in a second machine had obscured the issue, which was that lilo was attempting to mount and boot a root partition on /dev/hda10, a non-existent partition. Hm. So I put the drive back in angel, booted from a rescue disc, mounted the proper root partition, changed lilo.conf and reran lilo.

And it was perfect. Worked fine. No disc problem at all.

What this means is that at some point I must have edited lilo.conf, set the root partition to be /dev/hda10, reran lilo, and then not rebooted the machine to test it. Even for me this is shocking ineptitude. I have no idea why I did it. I must have been possessed by the spirit of a total fuckwit or something, that day.

You know the feeling of looking back on stuff you've written; an old school yearbook, perhaps, or old Usenet posts, and you cringe and realise what an idiot you were and wonder whether everyone else thought that at the time and never mentioned it? Yeah. Just like that. Oof.

So, anyway, angel is back, along with the other sites hosted on it. To those other guys, I can only grovel apologetically. Sorry, dudes. I'll try and do better next time. I'd promise to never do something this stupid again, but I'd be lying. I'm bound to do something this stupid again. Probably more stupid. Learning from mistakes is all well and good, but learning to not make any mistakes is pretty much impossible, I think.

And, sorry, gentle reader, if you were deprived of as days pass by for a bit. Only a few days passed by, so it wasn't too bad.

We now return you to your regularly scheduled service.

© sil, August 2002