Tuesday, January 30, 2007

It Pays to be Paranoid

You know, sometimes it just really pays to be overly paranoid, esp about backups. Here's the deal...

I've got my home web server and database. I had an array controller start acting up on me about a month ago and I knew I was gonna have to replace it. So I started the process of making sure I had good backups of all my data in case the controller went out before I had a chance to fix it. I copied my DBs and websites to a different partition on the same box. I also backed them up to my NAS. I also copied them over to a different server. And just to be sure, in case the backup was corrupt (which is a real possibility with Windows backup), I copied the raw files to the NAS as well. OK, so I've got all of this paranoia going on here, but this is very important data to me and I just can't afford to lose it.

So I got my array back up and everything up and running again. No problems. Now, this weekend, I lost a drive on that array. And of course, I'm using Raid-0. Why? Well, because disk is expensive and this is my home system, and sometimes it's hard to justify the extra cost of making things redundant. Well, I've been running backups to the NAS pretty much every day, so I knew I was protected. Unfortunately, I found that my NAS had lost a disk too... and you guessed it... Raid-0 as well. OK, I could ordinarily go to that other server that I copied it to from the previous outage, but I'm in the middle of rebuilding that server for my new Exchange server. So that data's no longer there... great.

Now it looks like I've lost all my data. I can't get the array back, and I've lost something like 12yrs of data. It's a complete disaster. Well, last night after talking to one of those disk recovery folks, I was about to pack up my array and send it to them for the $3,000 data retrieval process, when I looked at the other partition on my server and there it was... the data I had copied over from the previous meltdown! I couldn't believe it. Furtunately, my data doesn't change often enough to really worry about a couple weeks going by between backups, so I haven't lost anything at all.

Let me just say that when it comes to backups, it pays to be extremely paranoid. If your data's that important, act like it. I thought that my NAS and other server would be good enough, but they clearly weren't. So always plan for a multiple failure if it's at all possible. You could have electrical problems that knock out more than one server, you could have a flood or other natural disaster, etc. I'm serious here. If the data you're protecting is that important, treat it as such. In fact, treat it like a newborn baby. Protect it and never let anything happen to it.

I'll definitely be revamping my backup strategy at home so nothing like this ever happens again. I guess the first part of that is to stop being so cheap and get off Raid-0. I just can't afford that anymore. So you guys learn from my almost misfortune and take a good hard look at your backups with a paranoid eye. Develop impossible scenarios and see if your strategy can overcome them.

Good Luck!

0 comments:

About Me

My Photo
Sean McCown
I am a Contributing Editor for InfoWorld Magazine, and a frequent contributor to SQLServerCentral.com as well as SSWUG.org. I live with my wife and 3 kids, and have practiced and taught Kenpo for 22yrs now.
View my complete profile

Labels

Blogumulus by Roy Tanck and Amanda Fazani

Page Views