Advertisement:

Author Topic: Recent Downtime  (Read 161816 times)

Offline dschwab9

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 5,320
  • Gender: Male
    • Technology, Ministry and Other Ramblings
Recent Downtime
« on: January 19, 2008, 02:02:07 AM »
As you all know, we just had quite a long downtime.  Just wanted to post an update to let you guys know what happened and assure you that we weren't hacked or anything like that.

Things started out as a higher than normal CPU load and slightly sluggish server and pretty quickly went south.  I arrived at the datacenter expecting a runaway process or something was making the machine unreachable.  Instead, what I found was a server that was hot enough to fry eggs on and a seriously corrupted file system.  Below is, best I can tell, the chain of events that occured:

1.  The thermal management system on the motherboard wasn't properly throttling fan speed, possibly for quit a while.  This caused the CPU temperature to hoover near the danger zone off approximately 65 degrees C.
2.  CPU load rose higher than normal, possibly caused be heavy traffic and/or a bug on the latest site upgrade.
3.  The increase in CPU load caused a proportional increase in heat being produced.
4.  The CPU temperature rose to over 75 degrees C, well into the critical range.
5.  The CPU became flakey and ultimately caused the entire box to come down, corrupting nearly every file that was open at the time.

Then, the attempted repair:
1.  Ran fsck on the disks, repaired numerous errors, and began to restore corrupted files.
2.  Site appears to be relatively stable in maintenance mode.   In preporation to come out of maintenance mode and call things "good", we rebooted the box to see if any more errors pop up.  A few more file system errors were found and repaired.
3.  Another reboot was performed to see if the box would come up cleanly.  This time, we are greated with thousand upon thousands of file system problems, many of which were unrecoverable.
4.  At this point, I made the decision to leave the datacenter, go home, and bring in another server to replace the entire machine.
5.  New machine was installed and all apps reconfigured.
6.  A couple of critical directories were not properly backed up, so the old server was disassembled, the drives mounted in another machine, and a reconstruction done on that data.

So, it has been a looonnggggg night.  In order to prevent future problems like this, the fan speed control has been disabled on all of the servers.

Offline shadow82x

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 11,279
  • Gender: Male
    • cmblaber on LinkedIn
Re: Recent Downtime
« Reply #1 on: January 19, 2008, 02:04:30 AM »
Great to see SMF is finally up again. Some hard work there eh Derek?

Also I suppose avatars were cleared?
Colin B
Former Spammer, Customize, & Support Team Member

Offline Aaron

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 6,711
  • Gender: Male
    • Aaronweb.net
Re: Recent Downtime
« Reply #2 on: January 19, 2008, 02:08:18 AM »
Wow Derek, you've certainly been busy. Great job on bringing us back this fast, considering circumstances. Thanks for all the effort! :)

Offline JayBachatero

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 19,562
  • Gender: Male
    • @jaycreations on Twitter
    • JayBachatero.com
Re: Recent Downtime
« Reply #3 on: January 19, 2008, 02:09:16 AM »
Just as a side note.  Due to the file corruption some attachments were lost.  Including mods and themes.  Mod authors please look over your mods and check to see if any files are missing.

If you experience any errors or anything just let us know and we will look into it.  We are still working on a few things like downloads.  The page should be back up later on tonight. :)

BTW THANKS DEREK.  YOU THE MAN :)
Follow me on Twitter

"HELP!!! I've fallen and I can't get up"
This moment has been brought to you by LifeAlert

Offline nick09

  • Full Member
  • ***
  • Posts: 429
Re: Recent Downtime
« Reply #4 on: January 19, 2008, 02:10:40 AM »
well I'm happy to see the site is back.

Offline EdwinK

  • Jr. Member
  • **
  • Posts: 370
  • Gender: Male
  • Free is always better
Re: Recent Downtime
« Reply #5 on: January 19, 2008, 02:21:38 AM »
Glad to see the site up again ;)

Offline LightningMk6

  • Newbie
  • *
  • Posts: 7
Re: Recent Downtime
« Reply #6 on: January 19, 2008, 02:29:20 AM »
Well done on getting the site up and running again.

Offline Deaks

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 22,465
  • Gender: Male
    • pouvik on GitHub
    • SMFReview
Re: Recent Downtime
« Reply #7 on: January 19, 2008, 02:29:34 AM »
As it was mentioned the Theme Site may be missing files from your themes, we have just checked themes and all that is missing is thumbnail images, if you have a theme on the Theme Site can we please ask that you update your theme with a new image where possible.  All archive files are still there and are downloading fine.

Offline The Tornado

  • Semi-Newbie
  • *
  • Posts: 30
  • Gender: Male
  • A.I. is every thing
    • IT Group Syria Official Website
Re: Recent Downtime
« Reply #8 on: January 19, 2008, 02:31:37 AM »
well
i was trying to open the site all the day
thanks GOD it's here again ....

but i saw that u v updated the forums to smf 2.0 Beta 2 !!
when can it be available for members to update their forums to this version ?
help yourself by helping others

IT Group Syria Official Website

Offline TheWrath!

  • Full Member
  • ***
  • Posts: 432
  • Gender: Male
Re: Recent Downtime
« Reply #9 on: January 19, 2008, 02:32:21 AM »
hoo-ah for smf. good jobs fixing the problem guys!

Offline Zetan

  • Jr. Member
  • **
  • Posts: 175
  • Gender: Male
  • TP Support Team
    • LAB Radio
Re: Recent Downtime
« Reply #10 on: January 19, 2008, 02:32:43 AM »
Good job on a hard mission.. just a heads up,
The Search seems to be broken.

Offline shadow82x

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 11,279
  • Gender: Male
    • cmblaber on LinkedIn
Re: Recent Downtime
« Reply #11 on: January 19, 2008, 02:34:56 AM »

but i saw that u v updated the forums to smf 2.0 Beta 2 !!
when can it be available for members to update their forums to this version ?

There is no set time. Its ready when its ready. Currently a testing beta is aviable to charter members. (http://www.simplemachines.org/charter/)
Colin B
Former Spammer, Customize, & Support Team Member

Offline Deaks

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 22,465
  • Gender: Male
    • pouvik on GitHub
    • SMFReview
Re: Recent Downtime
« Reply #12 on: January 19, 2008, 02:35:00 AM »
when can it be available for members to update their forums to this version ?

im afraid you will have to wait for a newer version of smf2 before its being released in public, also until its released as the stable it should not be used on a live environment

Offline gffb

  • Semi-Newbie
  • *
  • Posts: 86
Re: Recent Downtime
« Reply #13 on: January 19, 2008, 02:35:30 AM »
Nice to see you back bt when searching I get the following message.

The search API could not be found! Please contact the admin to check they have uploaded the correct files.

Online Gary

  • Sorceress's Knight
  • Customizer
  • SMF Super Hero
  • *
  • Posts: 18,280
  • Gender: Male
  • So this is the luck of the draw...
    • Gazmanafc on Facebook
    • garygadsdon on LinkedIn
    • @Gazmanafc on Twitter
    • The Bongo Comics Fan Forum
Re: Recent Downtime
« Reply #14 on: January 19, 2008, 02:36:05 AM »
There will be at least one more charter beta before a public release.
Gary M. Gadsdon
Do NOT PM me unless I say so

War of the Simpsons
Bongo Comics Fan Forum
Youtube Let's Plays

^ YT is changing monetisation policy, help reach 1000 sub threshold.

Offline nick09

  • Full Member
  • ***
  • Posts: 429
Re: Recent Downtime
« Reply #15 on: January 19, 2008, 02:36:47 AM »
Quote
when can it be available for members to update their forums to this version ?

months,years possible.

but the current 2.0 is not recommended to try on major boards only test ones.

besides you have to be a charter members to even try the beta.

Offline CiOooo

  • Jr. Member
  • **
  • Posts: 203
  • Gender: Male
    • Sizin PlatForumunuz
Re: Recent Downtime
« Reply #16 on: January 19, 2008, 02:38:53 AM »
how a loooong night for u and a long time for us ;)

nice to see you back ;)



Offline Cheryl619

  • Newbie
  • *
  • Posts: 1
Re: Recent Downtime
« Reply #17 on: January 18, 2008, 06:36:16 PM »
We Missed You Guys :)

Offline Eugeniu

  • Semi-Newbie
  • *
  • Posts: 38
Re: Recent Downtime
« Reply #18 on: January 18, 2008, 06:46:12 PM »
Ah, that sounds terrible :(. Not to mention buying a new server out of nowhere probably wasn't fun either...

Glad to see that SimpleMachines is back online :).

Offline Dude111

  • Jr. Member
  • **
  • Posts: 305
  • An Awesome Dude
Re: Recent Downtime
« Reply #19 on: January 18, 2008, 06:47:13 PM »
Glad the site is back :)

SMF is awesome!!