Time for a rant.

A few years ago I decided it was finally time to learn Linux after having used DOS, Windows, and the Mac OS for years. My plan of attack was to run my own domain – savagexi.com, complete with a website, blogs, mail server, DNS server and DHCP server. And if I’d ever find the time, MythTv.

Back then Fedora seemed like the best choice, and every year or so I upgrade the servers in the basement to the latest version. Upgrading Fedora always sucks, but my experience over the weekend warrants a big, resounding F.

When working on my own machines, I tend to go beyond flying by the seat-of-my-pants to wanton recklessness. There’s nothing quite like a nasty error message (disk failure, missing partition, broken boot loader, misconfigured X server etc.) to focus the mind and learn how things really work. Over the years, my reckless attitude has cost me only once, when a disk drive that was part of a Logical Volume gave up its soul when it screeched to a dreadful halt. And even then, I almost managed to rescue the data I needed off the remaining disk, finding out five minutes too late what I should have done instead of what I did. Since then, I’ve eschewed LVM and gone with nice, simple RAID 1 arrays (which means having 2 disks that mirror each other so if one breaks you can get your data from the other one) to at least provide a modicum of redundancy.

The impetus for upgrading this time around was spam. I’ve always heard how wonderful greylisting is, and after one too many emails about navigating the love canal with confidence, it was time to take action. But of course I ran into a roadblock – setting up greylisting on Fedora 6 using a program called PostGrey didn’t work because it conflicted with SELinux (see, I’m a glutton for punishment, using SELinux on a home network). Of course that took some doing to figure out, since Fedora 6 doesn’t bother to actually log a message about the problem. So after reading the Fedora 8 release notes about how PostGrey and SELinux are best of buddies, I decided it was worth the pain to upgrade the email server.

From past experience, I was under no illusion it would be easy. But little did I suspect just how dreadful it would be. I decided to do the upgrade using a network install since I don’t have a DVD burner (yeah, yeah), which means the bytes are downloaded on demand across the Internet. It actually works pretty well if you pick a fast mirror, such as facebook. But when things go wrong you have to stop the installation, reboot the machine, Google around a bit, fix whatever problem is, start the installation over and redownload the bytes. Remember the stop-reboot-fix-install sequence, I must have done it twenty times.

Day 1

Attempt #1. Things got off to a rousing start with Anaconda, the Fedora installer, complaining that the disk partitions on the two drives in the machine had to be labeled. Of course Anaconda should have just fixed the problem itself, but no, it is a remarkably unhelpful program.

Attempt #2. So stop-reboot-google around-fix the problem – and try again. This time Anaconda bitched about not finding any valid partitions, or in English, it couldn’t read the 2 hard-drives on the machine and thus couldn’t update them. Since I had just rebooted the machine, it stretched the imagination that Anaconda could be so dumb. But either way, back to the stop-reboot-fix-reboot-start cycle. Except this time there was no fix, since the machine booted just fine.

Attempt #3. Try again. This time I gave into Anaconda when it offered the choice of wiping the drives clean, and hit the next button. I then quickly decided that was a bad move, and hit the back button. No luck. Although the installation hadn’t started yet (I was on screen that was asking me some question I don’t remember), when I rebooted the machine I was greeted by the message GRUB. Mind you, not a grub prompt, just four capital letters that spelled GRUB. Ugh.

So it was now time to dig out the Fedora 6 rescue disk and run it. It couldn’t find any partitions either, and dumped me at a command prompt. From there I could run the ever exciting program fdisk, which let’s you manage the partitions on your disk. fdisk is a nice, easy to use program, but its living on the edge – one false move and you can easily delete your data. From fdisk I noted that the machine had two hard-drives, the first was 80GB and the second 60GB. I also saw that first drive (80GB) no longer had a partition table thanks to Anaconda. Working backwards, I recreated its partitions. That was easy to do, since the two drives are part of a RAID array and thus I assumed the first partition on the first disk should be 60GB.

Attempt #4. Reboot and …. get greeted by the every friendly GRUB message again.

Attempt #5. Reboot, but this time I hit the F12 key to open the Boot menu. I then noticed that the last choice in the boot menu was to start a utility disk, which miraculously opened to a grub prompt (it wasn’t until the next day I figured out how to run grub from the rescue disk, although I suspected it was possible). Of course I don’t know diddly about Grub, so it took another 30 minutes of Googling to figure out how to fix the problem (basically reinstall grub on the drive).

Attempt #6. This time, Anaconda had the decency to recognize my partitions and even offered me a chance to upgrade them. Hooray. Pushing my luck, I hit the next button, and watched Anaconda check the dependencies for all installed packages. 5%, 10%, 15%, 25%, 26%…and then nothing. Of course.

More Googling, and finally enlightenment. Turns out I was hardly the first to run into this show-stopper bug. If that wasn’t bad enough, the bug was still open 2 months after it was reported, and none of the mirrors had been updated (there has been a respin of the the Fedora 8 CDs, but its hardly useful if I can’t get to it). So I read through the whole thread, and in one of the comments a Fedora developer had posted a link to a “update image” on his website. After a bit of research, I figured out what an update image is and how to use it.

Attempt #7. If you don’t first succeed, try, try again. This time Anaconda got past the dependency checker, and amazingly enough finished. Success was near at hand. NOT.

Attempt #8. Reboot the machine and watch in horror as the dreaded GRUB message rears its ugly head.

So back to the rescue disk – which of course can’t mount any partitions ( wtf?) and spits me out to a linux prompt. Back to fdisk. And once again enlightenment – Disk #1 had once again lost its partition table. Fix it. Boy this is getting tedious.

Attempt #9. Surely things are fixed by now. Reboot. And then watch in amazement as the computer tries to load Fedora Core 6, spits out pages and pages of errors, and unceremoniously dumps me to a login prompt. Of course the login prompt doesn’t work. WTF?

Ah – my favorite pastime, loading the rescue disk. Try fdisk again, everything looks ok. So the next obvious thing is the RAID array is broken somehow. Go read about mdadm, which is the Linux program for creating and managing software RAID arrays. Using the wonders of Google, I found a very helpful article that explains how to rescue your RAID array. Following the instructions, I remount the array and discover that only Disk #2 is available. And then it dawns on me – somehow Anaconda only updated Drive #2, thus leaving Drive #1 with Fedora 6 in a very broken state. So a bit more Googling, and I learn how to re-add Disk #1 back into the array. And then nothing. Hmm. More Googling – how exactly do you know what a RAID array is doing?

That didn’t take long, and I stare in wonderment as something actually goes right – mdadm is happily resyncing Disk #1 with Disk #2 and says it will be done in a bit over an hour. At this point its 3:30 am, so I call it a day.

Day 2

Attempt #10. After a good night’s sleep, it was time for more fun. The RAID array had successfully fixed itself overnight, so crossing my fingers I rebooted the machine. My heart sunk when I was greeted with lines and lines of warnings about disk overflow errors. But wait, those were for the extra partition on Disk #1 (remember only the first 60GB are used in the RAID array, leaving 20GB free). Once the cruft had cleared, the machine managed to boot all the way to the Fedora 8 welcome screen. Hallelujah! Of course a fair bit was broken, including the DNS server, which meant at least a few hours in BIND hell (BIND and I simply don’t get along). But first things first.

However, I was worried about the disk overflow errors. For some reason, the kernel thought the 20GB partition was smaller that it really was. A bit of Googling turned up a couple of potential causes and solutions, but none worked. So back to fdisk. I figured the best course of action was to just delete the 2nd partition and recreate it.

Attempt #11. After recreating the problem partition, it was time to reboot the machine. And of course back to my old friend GRUB. I have no idea how I ended up back there, but clearly old flings die slowly. But at this point I was an old hand at moving on, and rescue disk in hand, it was time to work some magic at the grub prompt. And to be on the safe-side, I Googled around a bit more to see if somehow I had mistakenly configured GRUB with RAID and could kick this habit once and for all. Fortunately, I turned up this gem of an article and promptly changed things around based on its recommendations.

Attempt #12. And finally, one day later, a clean boot to Fedora 8 (minus of course BIND being unhappy).

Denouement

It beats me how any normal person manages to maintain their own Linux system – I only succeed through sheer determination and stubbornness. I realize that Fedora recommends a clean install with each new version, but to do that without losing your personal data and system configuration takes knowledge and effort beyond almost anyone who lives on this planet, including myself. So overall – I give Fedora an F for its horribly broken upgrade program.

And of course the kicker – PostGrey still doesn’t work with SELinux on Fedora 8. But at least in FC8 its polite enough to actually log an error. So anyone for creating and compiling their own policy files? Ah, I feel another rant coming along about SELinux.