>> WHAT HAPPENED TO SNOWEST on THURSDAY/FRIDAY????

christopher

Well-known member
Staff member
Lifetime Membership
Nov 1, 2008
75,704
20,526
113
Rigby, Idaho
>>> WHAT HAPPENED TO SNOWEST ????
<hr style="color:#e8e8e8; background-color:#e8e8e8" size="1">



Thursday morning one of our hard drives crashed and took SnoWest offline. This in and of itself is no big deal. We built our system to be fault tolerant and the loss of any one drive only causes a minor hiccup. If everything works properly you never even know it happened!. When the servers went down I drove into town to check on them and we quickly realized that one of the drives had failed. I pulled that one out and did a hot swap replacement with a new drive we keep on hand for just such an emergency. When that new drive was installed we expected to bring SnoWest back online. But that didn't happen.



Yesterday when the first drive crashed and "burned out" a 2nd drive powered down for no explainable reason. That second drive shutting itself off, caused a MASSIVE loss of data in our drive array that took us about 19 hours to resolve. This meant that Brian had to make a 3 hour journey north to come and help get SnoWest back online once again. In the end he had to do a total rebuild of the entire array and that kept us offline till 3am Friday morning.

By the end of this weekend we will have replaced all 5 of our primary hard drives with brad new one, and we are going to add in 4 additional ones to ensure this scenario can NOT repeat itself ever again!



I apologize for the loss of service over the last 24 hours.
We are working hard to see to it that it never happens again!


My new desktop Paperweight!

__________________
__________________
 

christopher

Well-known member
Staff member
Lifetime Membership
Nov 1, 2008
75,704
20,526
113
Rigby, Idaho
And if that wasn't BAD ENOUGH.
We got slamed again this morning!

http://www.eastidahonews.com/2013/05/power-outage-in-idaho-falls/




(IDAHO FALLS, ID) – A major power outage has affected the Northwest section of Idaho Falls.

Around 9:52 this morning, a main circuit tripped that feeds three substations that affects Downtown Idaho Falls, and areas to the immediate North and West. A spokesman for Idaho Falls Power said they tried to return power down the line, but it did not hold.

Work crews are in the process of trying to isolate where the interference occurred and are working to restore power. So far there has been no indication as to how long the power might be out.
 

Excalibur

Well-known member
Lifetime Membership
Jan 29, 2011
404
165
43
Lewiston, ID
Thanks for the update and thank you guys for getting on it ASAP. We had a server hard drive failure at the shop 3 weeks ago, a bit stressful:face-icon-small-coo
 

K2

Well-known member
Dec 5, 2011
164
61
28
Anchorage AK
>>> WHAT HAPPENED TO SNOWEST ????
<hr style="color:#e8e8e8; background-color:#e8e8e8" size="1">



Thursday morning one of our hard drives crashed and took SnoWest offline. This in and of itself is no big deal. We built our system to be fault tolerant and the loss of any one drive only causes a minor hiccup. If everything works properly you never even know it happened!. When the servers went down I drove into town to check on them and we quickly realized that one of the drives had failed. I pulled that one out and did a hot swap replacement with a new drive we keep on hand for just such an emergency. When that new drive was installed we expected to bring SnoWest back online. But that didn't happen.



Yesterday when the first drive crashed and "burned out" a 2nd drive powered down for no explainable reason. That second drive shutting itself off, caused a MASSIVE loss of data in our drive array that took us about 19 hours to resolve. This meant that Brian had to make a 3 hour journey north to come and help get SnoWest back online once again. In the end he had to do a total rebuild of the entire array and that kept us offline till 3am Friday morning.

By the end of this weekend we will have replaced all 5 of our primary hard drives with brad new one, and we are going to add in 4 additional ones to ensure this scenario can NOT repeat itself ever again!



I apologize for the loss of service over the last 24 hours.
We are working hard to see to it that it never happens again!


My new desktop Paperweight!

__________________
__________________
is that thing dead ?

:face-icon-small-win
 

newmy1

Well-known member
Lifetime Membership
Sep 3, 2008
3,244
1,143
113
Huskerland
Nice work. Sound like a good site to outsource the hosting.


Sent from my iPhone using Tapatalk
 

Laundryboy

Well-known member
Lifetime Membership
Nov 1, 2012
1,247
312
83
One thing I remember about Idaho Falls when I lived there 20 years ago, was that the power would go out about once a week.
 

christopher

Well-known member
Staff member
Lifetime Membership
Nov 1, 2008
75,704
20,526
113
Rigby, Idaho
One thing I remember about Idaho Falls when I lived there 20 years ago, was that the power would go out about once a week.
Ya, it can be a challenge.
Our backups keep us going through anything small, but when the city goes down then our ISP goes down and that kills us.

Once we lost the fiber connection this morning, we were toast.
 

polrmk

Member
Premium Member
Dec 6, 2007
36
6
8
Goodhue, MN
We had a similar situation at work several years ago. We had several disk drives that mirrored each other. One went down in each section. It took about 2 days to get it running with IBM techs from Rochester working on it and a taxi bringing new hard drives from Chicago. Very short nights and a lot of stress.
 
Premium Features