ISP Issue

[Scott Greczkowski]

Just FYI I am hearing reports of issues getting to our servers. It appears there is an issue just outside our ISP.

I have contacted our ISP and they are aware of the issue and working to resolve it.

Thanks for your understanding!

 

[Scott Greczkowski]

Just an update from our ISP... Looks like things are fixed.

An FIB TCAM exhaustion issue was experienced this morning on the Premium Network. The initial incident resulted in a small number of routes entering a software-switched state. However, as the morning went on, this issue cascaded and required rolling reboots of our border routers.

At this time, the issue has been resolved.

 

[KAB]

Glad it's fixed. This is what I kept getting:

Fatal error: Memcache is not installed in [path]/includes/class_datastore.php on line 183

 

[rv1pop]

Glad it's fixed. This is what I kept getting:

Fatal error: Memcache is not installed in [path]/includes/class_datastore.php on line 183

Still getting same thing as have for 2+ hours
Fatal error: Memcache is not installed in [path]/includes/class_datastore.php on line 183

 

[Scott Greczkowski]

Are you still getting it now?

 

[wallyhts]

"
An FIB TCAM exhaustion issue was experienced this morning on the Premium Network. The initial incident resulted in a small number of routes entering a software-switched state. However, as the morning went on, this issue cascaded and required rolling reboots of our border routers.

At this time, the issue has been resolved."

What kind of place has rolling reboots of border routers?

Sounds like then need someone who can configure their catalyst correctly in my opinion.

 

[rv1pop]

Are you still getting it now?

seems to be OK now. Has not come up for 15 minutes or so
Thanks
POP

ps I guess that means I am on the border -- Central Georgia today.

 

[Scott Greczkowski]

What kind of place has rolling reboots of border routers?

I hear they use some of WIlly Wanka's Oompa Loompas to do it!

 

[wallyhts]

I hear they use some of WIlly Wanka's Oompa Loompas to do it!

LOL! That would be a Kodak moment for sure

 

[Scott Greczkowski]

Just a note that we have another event today (not related to the work being done on the servers.)

COLO4 UNPLANNED NETWORK EVENT NOTIFICATION:
________________________________________________________________

EVENT ID: COLO402132011-911-A

DATE: 02/13/2011

EVENT START TIME: 12:00 PM CST

EVENT END TIME: 1:00 PM CST

SERVICES/EQUIPMENT: Colo4 Premium Network

TYPE OF EVENT: Network Hardware Performance Incident

IMPACT OF EVENT: Reduced throughput, increased latency, service interruption

ACTION TAKEN: EMERGENCY REBOOT OF BORDER ROUTERS, router reconfiguration.

RESOLVED?: YES


Event Description:

At approximately noon today, an FIB TCAM exhaustion event resulted in the need to perform an emergency rolling reboot of each of our Premium Network border routers. This resulted in a period of reduced network capacity and network interruption.

We have identified the underlying cause of this issue and corrected it by reallocating resources on the routers in question and modifying the configuration affecting the distribution of these resources.

Please let us know if you are experiencing any issues at this time. At this time, the issue is believed to be fully resolved.

This notification is being sent to ALL CUSTOMERS. This notification pertains only to the PREMIUM NETWORK. The Value Network product was NOT AFFECTED.

If you have any questions or concerns please address them to our NOC at support@colo4.com or give us a call at 214-630-3100, and reference event number COLO402132011-911-A.