Update on Thursdays Server Problems

[Scott Greczkowski]

Hey guys, once again sorry about the server problems on Thursday, as you may know I was with my wife at a Wendy's Managers convention in the Bahamas.

I was able to fix the problem thanks to a phone call from Ilya to my hotel room, normally when there are server problems I am paged automaticlly by our server monitoring service. However in the Bahamas neither my pager nor my cell phone worked (My cell phone is Digital Only and in the Bahamas it is Analog only)

I have gone through the logs and have determined the reason for the problems was a process got stuck and kept restarting (without releasing the system resouces the old process was using) because of this all the system resouces were drained to the point the server just stopped working. This is why it was getting slower and slower all day then just stopped working.

Normally something like this would not happen, I usually monitor the server myself plus again I have a monitoring service which lets me know when things are starting to act funny, (most of the time I am able to fix problems before you guys even see them)

I do appologize for the inconvience. The good news is we have no travel plans until January for the CES show in Vegas.

 

[korsjs]

Hope you guys had fun.

 

[Scott Greczkowski]

Actually I didn't care for it much. We stayed at the Atlantis Royal Towers, and I was not impressed with the rooms or the service.

I am not a big drinker and it seemed most activities revolved around getting drunk.

We had a good time when we rented JetSkis (although we flipped over twice) The bad part was the salt water washed off our sunscreen and both me and my wife looked like lobsters.

The comedy club was also good. And we did OK at the casino, I figured out a methor to play the slots and not lose any money. My wife was shocked I kept going from machine to machine with $20 and in a few seconds had $40.

Wendys is talking about holding next years conference in Mexico.

 

[korsjs]

I know what you mean by most of the activities revole around alcohol. When I was in the Navy and visted the likes of Thailand, Bali, Singapore, Australlia, Philippines, Korea to name a few, the first thing most people did, myself included, after getting off the boat was head to the first street vendor selling cold ones. However, the Morale Welfare and Recreation (MWR) always had tour/activities set up for a very good price. We were not aloud to rent car/jet skis, unless it was a MWR sponsored event, we did anyway. I would have to say my favorite place to visit was Bali. We were one of the last ships authorized to visit Indonesia before the US put travel restrictions on them.

 

[TuxCoder]

And we did OK at the casino, I figured out a methor to play the slots and not lose any money. My wife was shocked I kept going from machine to machine with $20 and in a few seconds had $40.

Were you scooting your shoes on the floor and touching the machine's chassis at just the right time?

 

[Neutron]

What OS does our server use?

 

[TuxCoder]

What OS does our server use?

boston:~> telnet www.satelliteguys.us 80
Trying 67.19.94.137...
Connected to satelliteguys.us.
Escape character is '^]'.
HEAD / HTTP/1.0

HTTP/1.1 200 OK
Date: Sun, 10 Oct 2004 18:20:28 GMT
Server: [B]Apache/1.3.31 (Unix)[/B] mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.3.8 mod_ssl/2.8.19 OpenSSL/0.9.7a
X-Powered-By: PHP/4.3.8
Set-Cookie: bbsessionhash=f4239047a58e39138b4d332cd815fb42; path=/
Set-Cookie: bblastvisit=1097432450; expires=Mon, 10-Oct-05 18:20:28 GMT; path=/
Set-Cookie: bblastactivity=1097432450; expires=Mon, 10-Oct-05 18:20:28 GMT; path=/
Expires: 0
Cache-Control: private, post-check=0, pre-check=0, max-age=0
Pragma: no-cache
Connection: close
Content-Type: text/html; charset=ISO-8859-1

Connection closed by foreign host.
boston:~>

Some Unix variant running Apache.

 

[Neutron]

Is there a way to institute some code that will automatically reboot the server if the resources get below a certain level?

 

[TuxCoder]

Since it's a Unix, of course such a thing is doable!

And netcraft confirms my suspicion, it's running Linux!

 

[Neutron]

Which distro?

 

[Scott Greczkowski]

Yup we are running off of RedHat 9.0

 

[TuxCoder]

Scott, I hope you removed their crippled attempt at Unicode support... unless something on here really needs it? It f**ks up the display of all 'man' pages, not to mention slowing many things down. At work I made up a customized distribution that boots in 14 seconds (on a regular PC, even) and had I not removed the Unicode LANG settings it took significantly longer to start up.

What appalls me most is the man page output... RH must have just been ignoring that obvious problem. On an uncorrected system you must 'unset LANG' for every session to view them.

 

[Scott Greczkowski]

I have been through the system and have tweaked it to my liking.

We were lucky too have this server for a few weeks before it went on line, I was really able to work on it and get things tweaked nicely.

We do really well on this server too with 1,200 users on line our server load is only 0.17

 

[Neutron]

I would hate to see what would happen if the server was running Microsoft Windows....

 

[dfergie]

I would hate to see what would happen if the server was running Microsoft Windows....

or Commodore 64 Basic....