rssLink RSS for all categories
 
icon_red
icon_green
icon_red
icon_red
icon_blue
icon_green
icon_green
icon_red
icon_red
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_green
icon_blue
icon_green
icon_orange
icon_red
icon_green
icon_red
icon_red
icon_green
icon_red
icon_red
icon_red
icon_red
icon_orange
icon_green
 

FS#1299 — FS#5300 — pdc1-1-c1

Attached to Project— Network
Incident
Whole Network
CLOSED
100%
We have a problem at the level of the hard for this router.
Date:  Sunday, 03 April 2011, 01:32AM
Reason for closing:  Done
Comment by OVH - Saturday, 02 April 2011, 18:30PM

Summary of the actions that we did so far:
The router runs currently on the card #1. We tried to replace the card #2 that was crashed and displayed by the router as a default.
As we insert the card of the spare in the slot #2 , the chassis bloks againa. So, we restarted it in hard only on the card #1 in order to have a cold restart of card #1.

After this restarting, new trial of insertion of the spare card in slot #2 , the whole ports of the chassis are in default (!). We take off this time only the card #2 and we have a normal functionning.

We suspect the card #1 to be the origin of the problem even though it works currently in standalone.
A new reboot hard of the chassis will be necessary to restart on the card #2. We will restart after that the card #1 by a spare.


Comment by OVH - Saturday, 02 April 2011, 18:35PM

We are doing the hard reboot of the chassis in 30 minutes. Another incident is in progress on p19 (#5301).
The chassis works now in degraded mode but it is stable.


Comment by OVH - Saturday, 02 April 2011, 18:35PM

We are doing the reboot to have a cold restart of card #2.


Comment by OVH - Saturday, 02 April 2011, 18:53PM

Neither the original card #2 nor the spare card does restart in the slot1 or 2. We are reinserting the original card #1.


Comment by OVH - Saturday, 02 April 2011, 19:22PM

Impossible to start any card on the chassis (!).
We are preparing ourselves to replace the whole chassis urgently.


Comment by OVH - Saturday, 02 April 2011, 19:39PM

After investigation, it turned out that we are dealing with different
crashes of different levels rather than with a chassis problem.
The chassis was booted on a new spare. We are downgrading the configuration
of the chassis manually.


Comment by OVH - Saturday, 02 April 2011, 21:48PM

The chassis is up on a new card #1. The configuration is synchronised.
We are going to test the precedent cards in lab and prepare a new card #2
from scratch.

We will intervene most probably during the night in order
to reinsert the card #2.


Comment by OVH - Sunday, 03 April 2011, 00:30AM

We are going to intervene within few minutes to reinsert a new card on the slot #2. Probably the chassis crash is the problem's origin. Since we have been able to boot the new card without problem #1, we had good reasons to hope that it would be no problem now. Nevertheless, at this stage and regarding the problems that we faced, we could not be ensure anything. If the chassis is the problem's origin, we'll replace it.


Comment by OVH - Sunday, 03 April 2011, 00:36AM

The new card #2 has booted successfully. We re-descend the setting of the routing module m2.


Comment by OVH - Sunday, 03 April 2011, 01:31AM

The router is now back to the normal status.

The chassis itself seems out of the cause but we failed to reproduce the boot problem of origin sups cards (and of the 1st spare card) on the test chassis.
It is probable in any case that we'd cumulate many problems at a time.

We do apologize for this downtime of our customers on pdc1-1 in 1x2. Clients in 2x2 or 2x4 have not been impacted.