The server behind getgnulinux.org experienced some troubles today. For some reason the server had rebooted and the GRUB configuration file was gone, welcoming me with the message,
error: file not found
grub rescue>
So it was no longer able to boot into the operating system. Not knowing how to use the grub command-line, I used the rescue mode of the Ubuntu server CD to re-generate the GRUB configuration file. After rebooting the system it seems that everything is working again. All data still seems to be intact and the websites are back up. I have not found any indications of the server being cracked, so I have no idea what caused this.
Update 2012/03/13: I found the cause of the problem. This issue was caused by an external hard disk drive that is permanently connected to the server via USB. Backups are automatically written to this external hard drive. The server’s BIOS was setup to boot from USB devices first and so the server tries to boot from this external drive. However, the boot fails because GRUB was looking in this drive for a GRUB configuration file (which isn’t there). What was GRUB doing on that external drive? Well, that is a leftover from a previous Ubuntu installation (GRUB was installed on the MBR). So I deleted GRUB from the MBR with,
dd if=/dev/zero of=/dev/sdb bs=446 count=1
Of course “/dev/sdb” is the external drive. But after this the server still tried booting from the external drive, leaving me with a black screen and a blinking underscore at boot time. *sigh* Why is the BIOS still trying to boot from this external drive when GRUB is removed from the MBR?
Ultimately I had to solve work around this issue by just setting in the BIOS to not boot from USB devices.