First Outage on New Server

[Scott Greczkowski]

This server has been running great but woke up to a concerning email.

So I went in and checked it out. Did a database check and it confirmed we had some database corruption.

ran a repair on the database and then poof SatelliteGuys was down showing a database error.

I checked and repaired the database again but told me there were no issues yet we were still down.

i also did a restore of yesterdays database meaning we would have lost up to one days worth of posts.

Before I did that I said F it and let me reboot the server. So the server rebooted.

Glad I did that as everything came back and is testing fine and we didn’t lose a single post.

*phew!*

looking at the logs it appears the issue was caused by having two backups going at once which locked those tables.

We actually run three backups… Cpanel does a complete backup nightly, JetBackup does a backup every 6 hours. And then I do a Xenforo database backup at 3 am.

Cpanel backups are stored locally on the server and then copied to a FTP server at home.

Jetbackups are backed up to AWS via S3.

and the Xenforo Backups are stored locally and backed up to another server in Chicago.

i will need to update the Cpanel backup time so it does not conflict with the Jetbackup jobs.

so sorry we’re were down for a few minutes but so happy I was able to recover from it quickly.

 

[raoul5788]

It must have given you a bit of a start!

 

[navychop]

Better than coffee?

 

[TheKrell]

All of a sudden, I cannot toggle the sidebar off at the beginning of a new session. All subsequent page requests, or even just a refresh, results in the sidebar returning.

Any ideas on this one?

 

[Scott Greczkowski]

Nope no changes have been made. Sounds like a cookie issue as that is where that is stored.

 

[vazhog]

Welcome to the New Server --Same as the old server

 

[TheKrell]

Nope no changes have been made. Sounds like a cookie issue as that is where that is stored.

Unfortunately, clearing all SG cookies didn't help. I'm at work right now, so perhaps our firewall is doing something weird.

 

[TheKrell]

Same deal on another computer, too. Let me try another browser... Huh. Chrome is doing the same thing. Firewall weirdness?

 

[Bobby]

I collapse my sidebar by default rather than using the toggle. I had similar issues previously. To use the default go to your Preferences under your User Name (top right corner). Under Style Options check Collapse Side Bar By Default.

 

[Scott Greczkowski]

There is an issue... I can not log in to the forums admin panel.

I am investigating now.

 

[Scott Greczkowski]

Ok I think I know what the issue is...

We have been working to move our ad inventory to Ezoic and it appears this afternoon they switched us on... and we are now being fed through their cloudflare server and the caching on their server is too agressive and is causing the issue the kell is seeing as well as me not being able to log into the admin control panel.

I just turned Ezoic off but may take an hour for it to take effect.

 

[Scott Greczkowski]

And that indeed is what the problem is.

Now we wait for the Ezoic to turn off and things should be back to normal.

 

[navychop]

Never to return?

 

[Scott Greczkowski]

Never to return?

No it will but I just didn't have time to screw with their settings to fix it.
I set it up on Monday and it takes around 3 days for their setup to take effect. And it took effect this afternoon finally.

I will have to set it up again and this time tell it no to use their caching from Cloudflare. That should take care of the issue.
But for the site to survive it needs the ad revenue. (I dont like begging for money) lol.
Here is more about Ezoic. Everyone I know who have tried it are see good results (After the initial setup headaches)

https://www.ezoic.com/

 

[Scott Greczkowski]

Same deal on another computer, too. Let me try another browser... Huh. Chrome is doing the same thing. Firewall weirdness?

How’s it working now?

 

[raoul5788]

How’s it working now?

Everything seems normal to me.

 

[TheKrell]

Same deal on another computer, too. Let me try another browser... Huh. Chrome is doing the same thing. Firewall weirdness?

Works from home.

How’s it working now?

Oh. Yes, it seems to be fine as of Thursday morning. I'll try Bobby's default fix anyhow.

 

[Scott Greczkowski]

For those wondering why we are looking at Ezoic here is the reason.

https://xenforo.com/community/threads/xenforo-and-ezoic.215918/

We do need to figure out our ad issue.