OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#20149 — Telefonica (AS3352)
Incident Report for Network & Infrastructure
Resolved
Hello,
Among the services that we offer our customers, we include DDoS management which consists of receiving and cleaning DDoS attacks. To do this we have a large Internet capacity (7500Gbps) and we have developed VAC which makes it possible to clean DDoS. We have already received and successfully cleaned DDoS of 800Gbps.
For the past several days, we have been receiving an attack from a specific network, from the historic operator in Spain: Telefonica.
The attack targets a specific customer which is hosted in our BHS (Canada) datacenter.
It is not a DDoS (Distributed Denial of Service). It is not a DoS Denial of Service because it is not distributed. In fact, the attack doesn't come from many places around the world, but comes from a specific place and therefore uses a specific route.
AS3352 Telefonica Espagne <> AS12956 Telefonica International <> AS16276 OVH
We have linked the connections with AS12956:
30G + 20G in Madrid
40G in Paris
20G in Ashburn, VA
20G in Miami, FL
In total we have 130Gbps with AS12956 except that the DoS that we receive is 150Gbps.
Normally there is no problem to receive 150Gbps if it's a DDoS that comes from Asia, Europe and USA at the same time. Each piece of the network takes a part of the DDoS and each VAC cleans part of the DDoS. We have no problem to clean a basic DoS but in this particular case we cannot clean the DoS because we are not receiving it. The links that we have with Telefonica International are saturated before we can act.
Therefore we have cut the BGP announce with AS12956 in order to use the other links that we have to the Internet.
Thus, Traffic is arriving by AS5511 OpenTransit.
AS3352 Telefonica Espagne <> AS12956 Telefonica International <> AS5511 OpenTransit <> AS16276 OVH
We have 1x100G with OTI in Frankfurt. The attack has saturated the link that we have with OTI. During this saturation of other ISP in Spain we have been impacted because we use OTI to receive traffic originating from Oragne Spain, Jazznet, etc.
We have cut the BGP announce with AS5511 in order to use the other links we have with the Internet.
Traffic now comes from Level3 AS3356.
AS3352 Telefonica Espagne <> AS12956 Telefonica International <> AS3356 Level3 <> AS16276 OVH

We have a capacity of 800Gbps with Level3 and we have several other 200Gbps links. Therefore we can now receive this DoS of 150Gbps.
The issue between AS12956 <> AS3356 is that there is not enough capacity to pass the DDoS without saturating the links between these operators.
We continue to work to resolve this issue. We are in contact with AS12956 which has asked AS3352 to shut down the botnet network that is at the source of the DoS. Also, we are modifying our respective configurations to successfully pass the DoS between AS12956 and AS16276.
We do not want to give more details because this task will be read by the hackers responsible for this DoS and they'll want to use this information to evade the tricks that we deploy.
Also, we will not completer this task before this evening. In the case of DDoS, the less information given, the less excited the hackers - making it better to manage the DDoS.
Having seen the impact on our Spanish customers, we wanted to provide them with information on the origin of the issue.
We have found some tricks to pass this DoS without saturating the links. We are going to see in the next few hours if it holds.
In any case, we have discussed with Telefonica International about increasing the capacity with their network. We are going to deploy a new router in Madrid sooner than expected. We will install it in October instead of March. This will permit us, in October, to connect 200G with Telefonica, upgrade Espanix to 2x 200G, OpenTransit to 200G and have these same upgrades with Telia and Cogent.
At the same time, we are going to add other links with Telefonica, notably in Paris with 200G and Ashburn, VA with 200G. The hackers have found a soft spot in our network. We are fixing this as quickly as possible. This experience will allow us to improve the protection that we offer our customers by default with our services.
Best Regards,
Octave

Update(s):

Date: 2016-09-05 18:07:27 UTC
The settings that we have implemented seem to be working just fine. We are still receiving 2-hour DoS every 4 to 5 hours.
We are no longer saturating the links.

If you still encounter any issues, please tweet me @olesovhcom

Date: 2016-09-05 18:03:24 UTC
We are working with Telefonica AS12956
in order to upgrade 2x10G to 1x100G at Ashburn,VA
https://status.ovh.net/?do=details&id=14005
Posted Sep 05, 2016 - 17:44 UTC