[Resolved] Jupiter DNS/reboot . . . - Page 8 - Dotable

Aussie Bob · 15th May 2007, 09:21 AM

It's all organge dots there now.

It takes the data and refreshes every 5 minutes, so each time the page gets refreshed, all the servers don't get pinged for each page refresh.

Gerrit · 15th May 2007, 09:22 AM

Quote:

Originally Posted by Aussie Bob

Loads are coming back now that the server has settled.

Yes I agree, there'll be some soul searching and disecting events as to what happened here. I've had them relabel Jupiter server as Jupiter server. That was a silly oversight that never should have happened in the first place.

I wasn't going to mention this in this thread, but since we're discussing it, here's what I think that happened.

When the 'old' Jupiter was exchanged, the machine that was going to become Pluto, which was bought, racked and labelled already, was the replacement server for Jupiter.
Somehow nobody thought of relabelling the server.
Until today the machine didn't need any attention of a colo tech so the wrong label was never noticed...

Aussie Bob · 15th May 2007, 09:38 AM

You got it in one Gerrit. That's exactly what happened. We now have the correct labelling on Jupiter server, and also in the APC reboot console too. We would have picked this problem up earlier, but Jup's been humming along now since the switch, that we haven't needed to reboot from our console, or call for one of the colo techs to reboot from the cab.

Fourbit · 15th May 2007, 10:01 AM

I am always amazed at the way Bob can keep his head up on these kind of days.

I would just like to add an observation and I think it's probably the biggest problem with Jupiter may even be an issue that needs addressing with the others as well.

Bob has said it many times and really didn't even notice what he said.

Quote:

but Jup's been humming along

SOMEBODY!!! Teach the DARN THING the WORDS to the song!!

Paul

Gerrit · 15th May 2007, 10:09 AM

Quote:

Originally Posted by Aussie Bob

You got it in one Gerrit.

You can send the case of Foster's to my billing address

hal · 16th May 2007, 01:58 AM

Bob
Have you got any further info about why there was a problem with DNS? Do the logs reveal anything?
Thanks
Hal

Aussie Bob · 16th May 2007, 11:36 AM

Quote:

Originally Posted by hal

Bob
Have you got any further info about why there was a problem with DNS? Do the logs reveal anything?

Yeah, it was a file system corruption that caused BIND to fail, and needed a -y fsck to fix. /home is 183GB so it takes a while to run the -y fsck.

15th May 2007, 09:21 AM	#71 (permalink)
Aussie Bob Dotable Founder Join Date: Mar 2005 Location: Land down under Posts: 3,687	It's all organge dots there now. It takes the data and refreshes every 5 minutes, so each time the page gets refreshed, all the servers don't get pinged for each page refresh. __________________ • Dotable • "an island of normality in a sea of pimps and pirates" •

15th May 2007, 09:38 AM	#73 (permalink)
Aussie Bob Dotable Founder Join Date: Mar 2005 Location: Land down under Posts: 3,687	You got it in one Gerrit. That's exactly what happened. We now have the correct labelling on Jupiter server, and also in the APC reboot console too. We would have picked this problem up earlier, but Jup's been humming along now since the switch, that we haven't needed to reboot from our console, or call for one of the colo techs to reboot from the cab. __________________ • Dotable • "an island of normality in a sea of pimps and pirates" •

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

16th May 2007, 01:58 AM	#76 (permalink)
hal Member Join Date: May 2006 Posts: 46	Bob Have you got any further info about why there was a problem with DNS? Do the logs reveal anything? Thanks Hal