We are back!

[Scott Greczkowski]

Hey guys, as you might have noticed we were down for most of the day. But I am happy to report we are back online

The bad news is we lost all posts for the day.

The problem happened when I was testing out the newly released vBulletin 3.50 Beta 1 software. In setting it up I clicked on the wrong item and because of it it trashed the SatelliteGuys database.

SatelliteGuys is backed up twice a day so we were able to recover, although not as quickly as I would have liked. Our database is now almost 3 gigabytes in space, with the majority of it being made up of the search index. The main reason it took so long to recover was because of the search index. Because our backup backs up everything there is no way of telling it not to back up the search index.

Again all posts made today (Tuesday June 14th) are gone.

I do appologize for the inconvience, and thank you all for your patience.

 

[korsjs]

thanks for keeping us up to date.

 

[Scott Greczkowski]

I tried doing my best, I had no clue when the import was going to complete. I really didn't expect it to take as long as it did.

Again thanks for your patience.

 

[chaddux]

http://www.satelliteguys.us/showpost.php?p=521114&postcount=72

I warned you.

 

[mcchesney]

Testing in production is a bad idea. How about setting up a Dev server to test the new release on?

 

[Scott Greczkowski]

I wasnt testing in production. I was doing it on its own seperate host and IP address. My mistake was I was logged in as a root account instead of the account username for our beta site.

If I was logged in as the account username none of this would have happened.

 

[dfergie]

NP.... DBS talk, Dbs forums and Avs got a few extra hit's... (not to mention just plain ole surfin and a dvd)

 

[Neutron]

Glad to see us back online!!

I missed my Satelliteguys fix today. I did keep checking though.

 

[riffjim4069]

I hate when that happens!!! Anyway, glad to see the site back online.

 

[KE4EST]

No prob Scott we all make mistakes. Glad to see it back up though.

 

[BFG]

Goober

I got you a gift if you don't already have one of these

 

[Neutron]

root account = major disaster waiting to happen if you aren't careful.

 

[Sadoun]

Glad to see back online, Scott.

 

[TuxCoder]

Thou shalt not take the name of root in vain...
I'm guilty of over-using root at work, though, but my excuse is it's a lot harder to work with the kernel modules I develop when you're not root.

 

[GregN]

With Ultrium LTO you can restore 400GB in about 2.5 hours.

2.6GB per min



http://www1.us.dell.com/content/products/compare.aspx/tapebackup_external_scsi?c=us&cs=555&l=en&s=biz

 

[stevesmall]

i thought maybe you sold out to the dolans and maybe they wanted to ruin another good thing>>>sat guys

 

[Scott Greczkowski]

With Ultrium LTO you can restore 400GB in about 2.5 hours.

That wasnt the problem, the backup was restored here in under two minutes (we have an offsite backup)

The problem was the amount of time it took to import the database back in to MySQL again. It seems that the bottle neck was our RAID 5 SCSI setup. Because during the import the cpu was not being taked at all, but the io wait to the hard drives was up around 97%.

 

[Mark_AR]

I was hoping it wasn't something like blackviper.com. :shocked There one day and gone the next. I really miss his XP tweaks.

My heart skipped a few beats when SGUS didn't come up!

 

[Scott Greczkowski]

I had to change the public_html directory to public_old and creat a new public_html directory which housed the message about the outtage, as I was getting HUNDREDS of Database error emails a min!

Glad all is well again.

 

[kanebear]

Been there, done that, got the T-shirt. My moment in infamy was wiping out a SCO box in one shot. Like you; logged in as root. I don't know what particular synapse misfired but rather than typing rm -r ./star400/log I just whacked in RM -R and nailed the enter key... from the root directory. Oops. Lucky me the box had been decommissioned but that was NOT the way I'd wanted to reconfigure it.