I’ve written here numerous dull tirades on the subject of backups. Well, here’s more.
We had my brand new shiny backup script working on the LAN to backup all the servers to a Linux box with a 300GB hard drive. For extra security, we copied it out to a Buffalo Terastation, which also serves as our office fileserver. For that extra bit of security, the Terastation is formatted as two shares, each of 250GB (using RAID-1). One of those shares is the office fileshare, the other is for server backups.
Well, there was a slight *cough* stupid *cough* problem with one of my backup scripts over a weekend, which resulted in a recursive backup of a directory (doh!). This filled up the disk on the Linux box, but it didn’t prevent it from happily trying to copy it all to the Terastation (using lftp).
When I came in on Monday, the Terastation was not happy. It simultaneously said the drives were ~30% full, and said that it couldn’t find any disks at all. FTP connections were dropped immediately. We were able to copy a few files off of it from machines that had kept the drive mounted via SMB, but then it would disconnect and vanish from that machine’s network visibility. This was not good. At some point, we thought it might be a good idea to try enabling another protocol to access the data, which had the unfortunate side effect of switching the Teraserver admin into Japanese.
Tech support took about 20 minutes to answer the call, but they were courteous and helpful. Eventually, they concluded that the controller board was bad. To get a replacement, they charged our credit card the price of a new unit, and shipped it out, with the understanding that we’d swap the drives into the new unit, send back the old one, and get credited back the money. While this is not ideal, I can understand why they do it that way.
In any case, the new unit arrived today. I went through the effort of swapping the drives from one unit to the other (which is a lot more complicated than it should be, requiring a lot of screws). And voila! Still a Japanese admin, and no ability to access the data.
My working theory now is that the Teraserver stores configuration data on the drives, and when the one share filled up, it corrupted the config data somehow. I’ll call tech support tomorrow and see what I can learn. *sigh*