OVHcloud Web Hosting Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#8281 — load balancing shared bis
Scheduled Maintenance Report for Web Cloud
Completed
Thereafter
http://status.ovh.net/?do=details&id=4339
we have contacted a10 and the setting that we are using.
There might be an improvement to apply to our
setting in order to better use the hardware that we have
in the box.

We are doing a test tonight for about 10 minutes
to see if the CPU is again loaded. If all went OK,
we will remove the routing then reset it tomorrow morning when we have the attack.


Update(s):

Date: 2013-03-20 14:45:15 UTC
We are waiting for the feedback of the a10 support
to fix this problem.

Meaningwhile, we are back to the initial configuration.

End of the task :)

Date: 2013-03-20 14:44:15 UTC
okey , we found the origin of the problem. It's a type
of attack. We set up a wordaround to make these attacks
ineffective and we'll looking with a10 how to block
them simply


All the trafic is on the 2 boxes.

Date: 2013-03-20 10:11:19 UTC
The CPU keeps increasing.

p19-77-a10#sh cpu
Time: 10:17:36 CET Wed Mar 20 2013
1Sec 5Sec 10Sec 30Sec 60Sec
--------------------------------------------------------
Control1 3% 4% 5% 5% 5%
Data1 82% 80% 81% 78% 77%
Data2 73% 70% 70% 70% 69%
Data3 77% 75% 75% 71% 70%
Data4 73% 73% 72% 72% 70%
Data5 79% 80% 79% 75% 74%
Data6 75% 71% 70% 70% 71%
Data7 73% 74% 74% 74% 73%
Data8 57% 57% 59% 57% 56%
Data9 74% 75% 76% 75% 75%
Data10 68% 71% 72% 73% 72%
Data11 72% 72% 72% 71% 70%
Data12 71% 71% 68% 69% 68%
Data13 79% 80% 80% 76% 75%
Data14 77% 77% 76% 75% 72%
Data15 76% 75% 74% 77% 76%

Mar 20 2013 10:25:40 Warning [AX]:conn proxy queue depth exceeds
limit (9001)
Mar 20 2013 10:25:38 Warning [AX]:conn proxy queue depth exceeds limit
(8001)
Mar 20 2013 10:21:54 Warning [AX]:conn proxy queue depth exceeds limit
(7001)
Mar 20 2013 10:21:51 Warning [AX]:conn proxy queue depth exceeds limit
(6001)
Mar 20 2013 10:20:49 Warning [AX]:conn proxy queue depth exceeds limit
(5001)
Mar 20 2013 10:18:58 Warning [AX]:conn proxy queue depth exceeds limit
(4001)
Mar 20 2013 10:18:57 Warning [AX]:conn proxy queue depth exceeds limit
(3001)
Mar 20 2013 10:04:29 Warning [AX]:conn proxy queue depth exceeds limit
(2001)
Mar 20 2013 10:04:27 Warning [AX]:conn proxy queue depth exceeds limit
(1001)
Mar 20 2013 10:04:25 Warning [AX]:conn proxy queue depth exceeds limit
(1)


Date: 2013-03-20 10:10:30 UTC
Mar 20 2013 10:04:29 Warning [AX]:conn proxy queue depth exceeds
limit (2001)
Mar 20 2013 10:04:27 Warning [AX]:conn proxy queue depth exceeds limit
(1001)
Mar 20 2013 10:04:25 Warning [AX]:conn proxy queue depth exceeds limit
(1)

Date: 2013-03-20 10:09:44 UTC
We tried to switch all the traffic to AX. The devices are overloaded. We think about a solution.

We switch a part of the traffic and it works fine. We switch the rest and it doesn't work. We need to find a solution.

We modify the settings on buffers
slb buff-thresh hw-buff 30720 relieve-thresh 15360 sys-buff-low 165000 sys-buff-high 235000
Devices reboot

It works with one device but the buffer line has not resisted the reboot!? So it's not that.

Date: 2013-03-20 00:53:32 UTC
Removed. The 1st box seems working properly.
However, the 2nd is not taking connections well.
We'll see if there's a hardware issue with it.

Tomorrow morning, we will reset the traffic on the
1st box only and see if bears it all and if we really
have the attack planned in auto all mornings (that we did not see
as ACE are well bearing the attack).
Posted Mar 20, 2013 - 00:46 UTC