Lots of Database errors

[yaz96]

When trying to switch between pages, getting alot of Database Errors today.

And the site could not be found a few times over the past couple of days.

Thought I'd let you know.

 

[riffjim4069]

Yep, it's just like being at AVSForum...

 

[Scott Greczkowski]

The database errors were do to our ISP accidentally unplugging us from the database server when they were looking at something for me.

Since late Thursday night we have been having a problem with our web server. At times the site would just die and the screen would fill with page fault errors.

Larry suspected something wrong with one of our drives early on, but our RAID cards reported no problems... until this morning when it reported one of the drives had bad sectors on it. Now I knew which drive was going bad. I ordered a new drive and have it being sent overnight to Dallas and it will be replaced tomorrow.

Hopefully this fixes the issues we have been seeing. This is the second drive to fail on it this year. So much for server grade drives.

 

[Jimbo]

I've reporteed the same issues, mine started around 5, but I haven't been on too much today.

 

[Scott Greczkowski]

Yup the database errors started at 5 and were fixed as soon as I was notified of the issue.

The other issue as I said has been going on since Thursday night.

 

[sundude90]

Yup the database errors started at 5 and were fixed as soon as I was notified of the issue.

The other issue as I said has been going on since Thursday night.

I thought this site was on Cloud Servers?

I think you need a better raid server, seems like you have had tons of issues with it.

 

[Scott Greczkowski]

No only images are sent from the cloud servers.

The issue is not the raid server, the issue is the hard drives failing. They are mechanical they will fail. Got to love these "Server Grade" drives.

 

[mike123abc]

Need to switch to SSD!

 

[Scott Greczkowski]

They are not big enough yet.

Besides i has had 2 SSD drives fail on me due to controller board failure... They died faster then a standard drive.

 

[sundude90]

You should have a server that when a Hard Drive Fails, the whole thing doesn't start messing up. One should be able to die without causing any issues.

 

[Scott Greczkowski]

Ahh if the world was a perfect world that would be the case, but its not.

 

[rockymtnhigh]

You should have a server that when a Hard Drive Fails, the whole thing doesn't start messing up. One should be able to die without causing any issues.

It is all about the benjamins.

 

[yaz96]

You should have a server that when a Hard Drive Fails, the whole thing doesn't start messing up. One should be able to die without causing any issues.

Feel free to pony up the dough! Do-nate......

 

[rockymtnhigh]

Ahh if the world was a perfect world that would be the case, but its not.

As one of my favorite movie characters once said, "Screws fall out all the time, the world is an imperfect place. ..."

 

[Hall]

Besides i has had 2 SSD drives fail on me due to controller board failure... They died faster then a standard drive.

And with a "mechanical" hard drive, you likely have signs that it's starting to fail. With an SSD, it probably just goes *poof* one day and it's gone.

 

[Scott Greczkowski]

Yup thats true Hall... (I use SSD drives on the computer in my truck, since the vibration o f the truck quickly killed my regular hard drives.)

With a big RAID 10 aray like we have we did not know which drive was bad until the raid card reported thats one of the drives was starting to have sector errors.

Until then we couldnt do anything as we did not know which drive to replace in the array.

 

[scooby2]

What kind of RAID card? Some allow you to check the predictive failure and SMART variables of the drives. I have some nice Nagios plugins to throw warnings when drives believe they are going to fail. When they start failing or completely fail it throws a critical.

 

[Scott Greczkowski]

They are 3WARE cards. (9500s I believe)

When the problem first started happening, LER suspected a drive or RAID issue because of all the Page Fault error we were getting. The problem was we didn't know what was wrong until a few day later when the RAID controller noticed the drive had bad sectors and was trying to fix it.

That is when we finally knew exactly what the problem was and what drive to replace.

 

[Foxbat]

With a big RAID 10 aray like we have we did not know which drive was bad until the raid card reported thats one of the drives was starting to have sector errors.

Until then we couldnt do anything as we did not know which drive to replace in the array.

Why RAID 10? I assume the RAID controller is striping mirrored sets of drives because the other way (mirroring stripe sets) is more likely to fail (I should know!) If you have enough room and money, you set up a Hot-swap Spare that can be used to replace a drive that starts throwing errors. Whatever the situation, one drive in a RAID array should never cause the system to "see" the error at the file or application level. Losing one drive in a RAID array (except for RAID-0) should still leave the system in a usable, albeit vulnerable state.

 

[Scott Greczkowski]

I will leave the why RAID 10 to LER.

Both servers only have 4 drive bays and all 4 are in use, no room for a spare.