rssLink RSS for all categories
 
icon_red
icon_green
icon_red
icon_red
icon_blue
icon_green
icon_green
icon_red
icon_red
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_green
icon_blue
icon_green
icon_orange
icon_red
icon_green
icon_red
icon_red
icon_green
icon_red
icon_red
icon_red
icon_red
icon_orange
icon_green
 

FS#3272 — FS#7221 — vss-2-6k

Attached to Project— Network
Incident
Whole Network
CLOSED
100%
The card 4 crashed.
4 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX

Date:  Tuesday, 11 September 2012, 13:03PM
Reason for closing:  Done
Comment by OVH - Sunday, 26 August 2012, 15:16PM

Aug 26 14:54:25 vss-2-6k.fr.eu 151891: Aug 26 13:54:00 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 4 set Off (Module Reset due to exception or user request)

Aug 26 14:56:00 vss-2-6k.fr.eu 151893: Aug 26 12:55:35.946: %SYS-DFC4-5-RESTART: System restarted --
Aug 26 14:56:08 vss-2-6k.fr.eu 151894: Aug 26 13:55:41 GMT: %DIAG-SP-6-RUN_MINIMUM: Module 4: Running Minimal Diagnostics...
Aug 26 14:56:47 vss-2-6k.fr.eu 151895: Aug 26 13:56:22 GMT: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 4 to recover from error: Linecard received system exception. Errcode =
Aug 26 14:56:47 vss-2-6k.fr.eu 151896: Aug 26 13:56:22 GMT: %OIR-SP-3-PWRCYCLE: Card in module 4, is being power-cycled 'Off (Module Reset due to exception or user request)'
Aug 26 14:56:49 vss-2-6k.fr.eu 151897: Aug 26 13:56:22 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 4 set Off (Module Reset due to exception or user request)


Aug 26 14:58:27 vss-2-6k.fr.eu 151898: Aug 26 12:58:02.843: %SYS-DFC4-5-RESTART: System restarted --
Aug 26 14:58:33 vss-2-6k.fr.eu 151899: Aug 26 13:58:08 GMT: %DIAG-SP-6-RUN_MINIMUM: Module 4: Running Minimal Diagnostics...
Aug 26 14:59:17 vss-2-6k.fr.eu 151900: Aug 26 13:58:51 GMT: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 4 to recover from error: Linecard received system exception. Errcode =
Aug 26 14:59:17 vss-2-6k.fr.eu 151901: Aug 26 13:58:51 GMT: %OIR-SP-3-PWRCYCLE: Card in module 4, is being power-cycled 'Off (Module Reset due to exception or user request)'
Aug 26 14:59:18 vss-2-6k.fr.eu 151902: Aug 26 13:58:51 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 4 set Off (Module Reset due to exception or user request)
Aug 26 15:01:04 vss-2-6k.fr.eu 151903: Aug 26 14:00:37 GMT: %DIAG-SP-6-RUN_MINIMUM: Module 4: Running Minimal Diagnostics...
Aug 26 15:01:04 vss-2-6k.fr.eu 151904: Aug 26 13:00:30.900: %SYS-DFC4-5-RESTART: System restarted --
Aug 26 15:01:49 vss-2-6k.fr.eu 151905: Aug 26 14:01:24 GMT: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 4 to recover from error: Linecard received system exception. Errcode =
Aug 26 15:01:49 vss-2-6k.fr.eu 151906: Aug 26 14:01:24 GMT: %OIR-SP-3-PWRCYCLE: Card in module 4, is being power-cycled 'Off (Module Reset due to exception or user request)'
Aug 26 15:01:50 vss-2-6k.fr.eu 151907: Aug 26 14:01:24 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 4 set Off (Module Reset due to exception or user request)


Aug 26 15:03:28 vss-2-6k.fr.eu 151908: Aug 26 14:03:03 GMT: %OIR-SP-6-REMCARD: Card removed from slot 4, interfaces disabled
Aug 26 15:05:02 vss-2-6k.fr.eu 151916: Aug 26 13:04:38.914: %SYS-DFC4-5-RESTART: System restarted --
Aug 26 15:05:12 vss-2-6k.fr.eu 151917: Aug 26 14:04:44 GMT: %DIAG-SP-6-RUN_MINIMUM: Module 4: Running Minimal Diagnostics...
Aug 26 15:05:54 vss-2-6k.fr.eu 151920: Aug 26 14:05:28 GMT: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 4 to recover from error: Linecard received system exception. Errcode =
Aug 26 15:05:54 vss-2-6k.fr.eu 151921: Aug 26 14:05:28 GMT: %OIR-SP-3-PWRCYCLE: Card in module 4, is being power-cycled 'Off (Module Reset due to exception or user request)'
Aug 26 15:05:54 vss-2-6k.fr.eu 151922: Aug 26 14:05:28 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 4 set Off (Module Reset due to exception or user request)



Aug 26 15:07:31 vss-2-6k.fr.eu 151926: Aug 26 13:07:07.060: %SYS-DFC4-5-RESTART: System restarted --

Aug 26 15:07:40 vss-2-6k.fr.eu 151928: Aug 26 14:07:12 GMT: %DIAG-SP-6-RUN_MINIMUM: Module 4: Running Minimal Diagnostics...
Aug 26 15:08:16 vss-2-6k.fr.eu 151929: Aug 26 14:07:50 GMT: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 4 to recover from error: Linecard received system exception. Errcode =
Aug 26 15:08:16 vss-2-6k.fr.eu 151930: Aug 26 14:07:50 GMT: %OIR-SP-3-PWRCYCLE: Card in module 4, is being power-cycled 'Off (Module Reset due to exception or user request)'
Aug 26 15:08:18 vss-2-6k.fr.eu 151931: Aug 26 14:07:50 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 4 set Off (Module Reset due to exception or user request)


Comment by OVH - Sunday, 26 August 2012, 15:17PM

Aug 26 15:13:35 vss-2-6k.fr.eu 151949: Aug 26 14:13:09 GMT: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 4 to recover from error: Linecard received system exception. Errcode =
Aug 26 15:13:35 vss-2-6k.fr.eu 151950: Aug 26 14:13:09 GMT: %OIR-SP-3-PWRCYCLE: Card in module 4, is being power-cycled 'Off (Module Reset due to exception or user request)'
Aug 26 15:13:35 vss-2-6k.fr.eu 151951: .Aug 26 14:13:09 GMT: %XDR-6-XDRIPCNOTIFY: Message not sent to slot 4/0 (4) because of IPC error queue flush. Disabling linecard. (Expected during linecard OIR or system reloads)
Aug 26 15:13:37 vss-2-6k.fr.eu 151952: Aug 26 14:13:09 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 4 set Off (Module Reset due to exception or user request)
Aug 26 15:13:47 vss-2-6k.fr.eu 151953: Aug 26 14:13:20 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 4 set off (admin request)


It's dead. We interrupt it electrically and then we search for a spare.


Comment by OVH - Sunday, 26 August 2012, 16:00PM

We are going to get back the card 8 of vss-2b (router of backup) to insert it instead of card 4 of vss-2 which is damaged.


Comment by OVH - Sunday, 26 August 2012, 17:20PM

The card is replaced.


Comment by OVH - Sunday, 26 August 2012, 17:30PM

We are still noticing anomalies on the networks routed via vss-2-6k. We are looking for the origin of the problem.


Comment by OVH - Sunday, 26 August 2012, 23:02PM

We notice ping problems on the networks connected to the replaced card and also losses on the other networks. We are ready to restart completely the chassis.


Comment by OVH - Sunday, 26 August 2012, 23:03PM

We restarted the chassis.


Comment by OVH - Sunday, 26 August 2012, 23:06PM

So, we progressively lost 3 cards on the router
one after the other dugin 2 hours.
It's very rare and it explains that there are
impacts during 2h30.

So
the card 4 is dead
the card 6 is dead
the card 8 is dead

That' too much.

We interrupted vss-2b-6k. We are running on vss-2-6k.
We are checking to get back the spare cards
until we receive them.


Comment by OVH - Sunday, 26 August 2012, 23:09PM

The router is stable.

In all we had many short interruptions during 2h30,
the necessary time to find the cards which are not good
and take them off.