So what happened? Enterprise downtime - Dotable

Nathan · 14th May 2008, 10:03 AM

Silence is not golden in this instance.

Quote:

Originally Posted by Aussie Bob

We're back!

I'll give some more info in a minute or two . . .

A minute or two is up. What happened? What has been done to ensure it wont happen again?

Aussie Bob · 14th May 2008, 10:44 AM

Hi Nathan,

Our first priority is to bring the server back, and we were focussed on that. There was no "silence" when I was made aware of the issue, and this was communicated in the enterprise forum.

What happened was enterprise dropped and our techs could not reboot it from the reboot console. So they assigned this to L3 which is myself and Steve and we were supposed to receive an sms from Alertra, but this sms did not arrive. I came on shift several hours later and to my horror to see the server being down, so tried to reboot it myself, but no go there. I posted the announcement, contacted the colo where we had them manually reboot the server. This still did not bring back the server, so the colo's remote hands hooked up a crash cart to the server and were having problems getting the server to load up. Each time a fsck was attempted, this would fail, and eventually the server was able to fsck properly and load up.

That's what happened. It's not pretty, but the truth is not always pretty. What's been done to ensure this does not happen again - well we can't prevent a server from dropping or having issues, but we can prevent us not knowing about it if a monitoring system fails to report it being down. So I've done something I should have done years ago, and setup another 3rd party monitoring account for all Dotable servers, instead of just relying on Alertra.

Nathan · 14th May 2008, 11:45 AM

Thank you.

What I meant by silence was the delay in your quote of more information in a few minutes etc.

I understand you were working on it, hence me waiting patiently until all was well again before asking what went wrong.

Good to hear/read you are taking more proactive measures with monitoring.

Thank you again for the efforts and thank you for getting the server up with no casualties (lost data etc).

My post above wasn't meant to be a critisism, rather a search for an explaination.

The truth may not pretty as you say, but keep us in the communication loop and good, bad or ugly, on average people seem to be happy.

Your constant post in the announcement forum was reassuring.

Aussie Bob · 14th May 2008, 11:53 AM

Quote:

Originally Posted by Nathan

. . . What I meant by silence was the delay in your quote of more information in a few minutes etc.

Yeah, sorry about that. I just got side tracked, which is not hard to do. I am accountable and can be called on it if I fall short of the mark.

14th May 2008, 10:44 AM	#2 (permalink)
Aussie Bob Dotable Founder Join Date: Mar 2005 Location: Land down under Posts: 3,687	Hi Nathan, Our first priority is to bring the server back, and we were focussed on that. There was no "silence" when I was made aware of the issue, and this was communicated in the enterprise forum. What happened was enterprise dropped and our techs could not reboot it from the reboot console. So they assigned this to L3 which is myself and Steve and we were supposed to receive an sms from Alertra, but this sms did not arrive. I came on shift several hours later and to my horror to see the server being down, so tried to reboot it myself, but no go there. I posted the announcement, contacted the colo where we had them manually reboot the server. This still did not bring back the server, so the colo's remote hands hooked up a crash cart to the server and were having problems getting the server to load up. Each time a fsck was attempted, this would fail, and eventually the server was able to fsck properly and load up. That's what happened. It's not pretty, but the truth is not always pretty. What's been done to ensure this does not happen again - well we can't prevent a server from dropping or having issues, but we can prevent us not knowing about it if a monitoring system fails to report it being down. So I've done something I should have done years ago, and setup another 3rd party monitoring account for all Dotable servers, instead of just relying on Alertra. __________________ • Dotable • "an island of normality in a sea of pimps and pirates" •

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

14th May 2008, 11:45 AM	#3 (permalink)
Nathan New Dotable member Join Date: Jan 2008 Posts: 4	Thank you. What I meant by silence was the delay in your quote of more information in a few minutes etc. I understand you were working on it, hence me waiting patiently until all was well again before asking what went wrong. Good to hear/read you are taking more proactive measures with monitoring. Thank you again for the efforts and thank you for getting the server up with no casualties (lost data etc). My post above wasn't meant to be a critisism, rather a search for an explaination. The truth may not pretty as you say, but keep us in the communication loop and good, bad or ugly, on average people seem to be happy. Your constant post in the announcement forum was reassuring.