OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#6729 — Switching ACE20 to ACE30
Scheduled Maintenance Report for Network & Infrastructure
Completed
UPDATE : The intervention will be replanned

--------------------------------------------








After the incidents on the first ACE couple (model ACE20), we are going to migrate all the contexts to a new generation ACE couple (ACE30).
The intervention will take place this Monday 14th of May starting from 23:00. The operation will be transparent for the redundant contexts (99% of the cases).


*** Details of the operation ***
1. switch of all the contexts on the ACE-1 of rbx-s1
2. complete back up of the ACE-1 of rbx-s2
2. isolation of the ACE-1 of rbx-s2
3. restarting of the new ACE (ACE-3) in rbx-s2 with the back up of the ACE-1
4. progressive switch of the contexts of the ACE-1 of rbx-s1 to the ACE-3 of rbx-s2
5. back up of the ACE-1 of rbx-s1
6. isolation of the ACE-1
7. restarting of the new ACE-3
8. repartition of the contexts on the 2 new ACE30

Update(s):

Date: 2012-05-20 01:49:03 UTC
It's up.

Date: 2012-05-20 01:48:44 UTC
We are establishing the 2nd ACE30. it's done. We're waiting.

Date: 2012-05-20 01:48:04 UTC
We have rebooted 2 new ACE30. Everything is fine.
Some basic checks. No problem.

We are cutting both old ACE20. Waiting 5 seconds.
Well, it is not cut. Okey, the s1 and s2 setting are not the same. Now, it is cut.

We are establishing ACE30 on s1. it responds to ping. websites are still inaccessible, we need to give MAc time to update it. We're waiting. Websites are up.

Date: 2012-05-19 20:39:33 UTC
We put the contexts on the original ACE in their original configuration.
Study in progress with Cisco.

Date: 2012-05-19 20:38:30 UTC
The ACE30 does not want to restart!

%ACE-2-199008: DC controller:Itasca reads from mailbox fifo when empty
%ACE-2-199008: DC controller:Itasca reads from mailbox fifo when empty
%ACE-2-199008: DC controller:Itasca reads from mailbox fifo when empty
%ACE-2-199008: DC controller:Itasca reads from mailbox fifo when empty
%ACE-1-727008: HA: Failed to send heartbeats to peer. Internal error: Failed to communicate to IXP
%ACE-2-199006: Orderly reload started at Sat May 19 21:52:26 2012 by System. Reload reason: NP 3 Failed : NP ME Hung


Date: 2012-05-19 20:37:56 UTC
Back up in progress of ACE-1 of rbx-s1 and then restarting of the new ACE30 (ACE-3)



Date: 2012-05-19 20:37:16 UTC
Due to the perturbations of this afternoon, we are going to do the switch today.
Right now all the contexts were switched to rbx-s2.

Date: 2012-05-19 20:36:27 UTC
We checked the incompatibel contexts one by one.
There is no obstacle to the switch which is planned on Monday May 21st starting from 23:00.

Date: 2012-05-15 05:46:57 UTC
All have been reset.
We are going to correct the contexts with a compatible setting then replan the maintenance.


Date: 2012-05-15 05:45:12 UTC
There are several contexts with a setting non compatible to the software version on ACE30.
We are going backwards (back to ACE20) time to correct the non valid settings.

*** DETAILS ***
In the previous versions, in a 'class map', it was possible to associate a command 'match any' with other types of commands 'match'.
In the current version (recent major version), this is not possible ('match' criteria will be ignored).





Date: 2012-05-14 21:45:19 UTC
The ACE-1 card of rbx-s1 is isolated. We are restarting the ACE30 with the good configuration.

Date: 2012-05-14 21:44:24 UTC
The contexts are all on rbx-s1.

Date: 2012-05-14 21:43:55 UTC
Let's go !
Posted May 14, 2012 - 16:12 UTC