<hr style="color:#e8e8e8; background-color:#e8e8e8" size="1">
Thursday morning one of our hard drives crashed and took SnoWest offline. This in and of itself is no big deal. We built our system to be fault tolerant and the loss of any one drive only causes a minor hiccup. If everything works properly you never even know it happened!. When the servers went down I drove into town to check on them and we quickly realized that one of the drives had failed. I pulled that one out and did a hot swap replacement with a new drive we keep on hand for just such an emergency. When that new drive was installed we expected to bring SnoWest back online. But that didn't happen.
Yesterday when the first drive crashed and "burned out" a 2nd drive powered down for no explainable reason. That second drive shutting itself off, caused a MASSIVE loss of data in our drive array that took us about 19 hours to resolve. This meant that Brian had to make a 3 hour journey north to come and help get SnoWest back online once again. In the end he had to do a total rebuild of the entire array and that kept us offline till 3am Friday morning.
By the end of this weekend we will have replaced all 5 of our primary hard drives with brad new one, and we are going to add in 4 additional ones to ensure this scenario can NOT repeat itself ever again!
I apologize for the loss of service over the last 24 hours.
We are working hard to see to it that it never happens again!
My new desktop Paperweight!