So at some point during the summer a drive died. I just happened to look at 3DM disk management one day and noticed a drive had dropped out of the array. Oops - how long has that been like that? It was, of course, inevitible but hey, RAID5 with hot spare so no worries.
I ordered a replacement drive.
While I was waiting for it to come in another drive died. So, I ordered a replacement for it as well so I could change them both at the same time since I knew damned well that it was not going to be just a simple operation.
This should have been the cue that it is time to power down the server so it can be repaired (duh). But no, I kept it running 'cause it is in use all the time.
Well, one day, the System Administrator decided that he needed to transfer some files to the degraded array. The partitions are set to Read Only so the user permissions have to be changed for the partition.
Ya know, I'd fire that guy if he wasn't so damned good looking.
Wouldn't you know it - the server crashed while the permissions were being changed. Geez, maybe I should take it offline now and drag down to the shop to repair.
Get it to the shop and sure enough the entire F partition is missing (2T partition about half full). Doh. Power it down and wait for the drive to come in. Hopefully, the data can be recovered.
About a week later, the second drive finally comes in. I swap them out and boot up expecting to see a rebuild message. Nope - the RAID card only sees the two new drives and none of the old ones - incomplete array. So I call 3Ware. They had a fibre cut and had intermittent phone operation so I lost the tech in the middle of the call.
Did I mention that while Iím working on this I have the fans disconnected? The stack gets mighty warm with no air blowing at it.
So while working on it a third drive failed.
That flushing sound was the sound of 3T of data going bye, bye.
Here we go again.
Obviously, stacking 12 hardrives one atop the other ain't gonna work.
I'm done screwing around with this. I called up my supplier and raised his eyebrows when I told him what to order for me.