[Resolved] Jupiter DNS/reboot . . . - Page 7 - Dotable

Aussie Bob · 15th May 2007, 09:34 AM

Quote:

Originally Posted by Gerrit

Sites resolve now.. pfeeww!

Yes, reports are coming in to that effect.

mikemee · 15th May 2007, 09:40 AM

Bob, when you've had a chance to have a cuppa and regroup, I'm sure I'm not the only one who'd like a short post-mortem on today. 6 1/2 hours is a long time with no DNS, and it seems like there was more than a 'small BIND issue' contributing to the downtime today.

Off the top of my head from today's thread, contributions were:

+ initial delay of an hour between DNS problems and a response from dotable
+ whatever was wrong with BIND (not usually a source of problems)
+ the ongoing problems with the colo (not the 1st time they've contributed to issues)
+ incorrect fsck command (admittedly a small contributor overall, but how does it get that way?)

Thanks for your efforts and honesty today. No rush on the above (e.g. tomorrow ok) but I would like to understand it. Gotta be honest, with all the jupiter problems since the beginning I'm seriously considering a switch back to httpme (where I may get equally unlucky with a server, but perhaps not the same colo problems). Thanks. Hope your day improves!

Shepherdboy · 15th May 2007, 09:44 AM

Thanks Bob for keeping us in the loop today.

Shepherdboy

dkg · 15th May 2007, 09:45 AM

Thanks Aussie Bob for a great job of communicating.

Maybe I'm wrong but it appears to me this outage could have been as little as 1.5-2 hours instead of the actual 6 hours. I do hope someone can at the colo center can learn from the mistakes made today.

dkg · 15th May 2007, 09:49 AM

Quote:

Originally Posted by mikemee

Off the top of my head from today's thread, contributions were:

+ initial delay of an hour between DNS problems and a response from dotable
+ whatever was wrong with BIND (not usually a source of problems)
+ the ongoing problems with the colo (not the 1st time they've contributed to issues)
+ incorrect fsck command (admittedly a small contributor overall, but how does it get that way?)

I would add:
+ unable to find the server due to poor documentation
+ tech believing the server was unresponsive because the console was redirected to a serial port

Both of these of colo issues and I tend to agree with you about Jupiter and the past problems.

Aussie Bob · 15th May 2007, 10:00 AM

Loads are coming back now that the server has settled.

Yes I agree, there'll be some soul searching and disecting events as to what happened here. I've had them relabel Jupiter server as Jupiter server. That was a silly oversight that never should have happened in the first place.

hal · 15th May 2007, 10:01 AM

I would also like to add that I did not see the status indicators change to red on the status page during the issues.
I have brought this up before.
Could someone check it please, Bob?
Thanks

Gerrit · 15th May 2007, 10:01 AM

Loads are still high, but we're back.
Time for a break here ..

Thanks for keeping us updated Bob!

Aussie Bob · 15th May 2007, 10:05 AM

Quote:

Originally Posted by hal

I would also like to add that I did not see the status indicators change to red on the status page during the issues.

I did. It was a line of red dots, that became organge dots when Jup was rebooted and back.

Gerrit · 15th May 2007, 10:17 AM

Quote:

Originally Posted by Aussie Bob

I did. It was a line of red dots, that became organge dots when Jup was rebooted and back.

The status page is not auto- refreshing, right?

I refreshed every now and then and saw the line of red dots for Jupiter.

15th May 2007, 09:45 AM	#64 (permalink)
dkg Northwoodsman Join Date: May 2006 Location: Northwoods of Maine Posts: 761	Thanks Aussie Bob for a great job of communicating. Maybe I'm wrong but it appears to me this outage could have been as little as 1.5-2 hours instead of the actual 6 hours. I do hope someone can at the colo center can learn from the mistakes made today. __________________ Dave G. Click here to learn about a free eNom reseller account!!! How about a nice cabin rental on Moosehead Lake in Northern Maine? Aussie Bob fan since May 2002

15th May 2007, 10:00 AM	#66 (permalink)
Aussie Bob Dotable Founder Join Date: Mar 2005 Location: Land down under Posts: 4,070	Loads are coming back now that the server has settled. Yes I agree, there'll be some soul searching and disecting events as to what happened here. I've had them relabel Jupiter server as Jupiter server. That was a silly oversight that never should have happened in the first place. __________________ Please review Dotable at the following review sites - HostJury.com \| WebHostingStuff.com

15th May 2007, 10:01 AM	#68 (permalink)
Gerrit E=mc² Join Date: May 2006 Location: Puerto Rico Posts: 445	Loads are still high, but we're back. Time for a break here .. Thanks for keeping us updated Bob! __________________ Gerrit SiriuS Website Design

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

15th May 2007, 09:40 AM	#62 (permalink)
mikemee New Dotable member Join Date: May 2006 Posts: 23	Bob, when you've had a chance to have a cuppa and regroup, I'm sure I'm not the only one who'd like a short post-mortem on today. 6 1/2 hours is a long time with no DNS, and it seems like there was more than a 'small BIND issue' contributing to the downtime today. Off the top of my head from today's thread, contributions were: + initial delay of an hour between DNS problems and a response from dotable + whatever was wrong with BIND (not usually a source of problems) + the ongoing problems with the colo (not the 1st time they've contributed to issues) + incorrect fsck command (admittedly a small contributor overall, but how does it get that way?) Thanks for your efforts and honesty today. No rush on the above (e.g. tomorrow ok) but I would like to understand it. Gotta be honest, with all the jupiter problems since the beginning I'm seriously considering a switch back to httpme (where I may get equally unlucky with a server, but perhaps not the same colo problems). Thanks. Hope your day improves!

15th May 2007, 09:44 AM	#63 (permalink)
Shepherdboy New Dotable member Join Date: Feb 2007 Posts: 6	Thanks Bob for keeping us in the loop today. Shepherdboy

15th May 2007, 10:01 AM	#67 (permalink)
hal Member Join Date: May 2006 Posts: 54	I would also like to add that I did not see the status indicators change to red on the status page during the issues. I have brought this up before. Could someone check it please, Bob? Thanks