> This system will allot network resources on a per-customer basis, creating a budget that, once exceeded, will prevent a customer's traffic from degrading the service for anyone else on the platform
How would this work practically? If a single client is overflowing the edge router queues you are kindof screwed already? Even if you dropped all packets from that client you would need to still process the packets to figure out what client they belong to before dropping the packets?
I guess you could somehow do some shuffle sharding where a single client belongs to a few IP prefixes and when that client misbehaves you withdraw those prefixes using BGP to essentially black hole the network routes for that client. If the shuffle sharding is done right only the problem client will have issues as other clients on the same prefixes will be sharded to other prefixes.
It's load shedding, but it's weighted towards people abusing their quota usually over some rolling weighted average. The benefit is that they are dropped immediately at the edge rather than holding sockets open or using compute/resources. It usually takes 30s-1m to kick in.
In this specific case, it wasn't requests from the client that caused overload. It was the responses to those requests. So Cloudflare can avoid sending responses, and prevent the problem.
You're right that this doesn't solve all cases, but it would have prevented this case.
> Even if you dropped all packets from that client you would need to still process the packets to figure out what client they belong to before dropping the packets?
In modern Linux you can write BPF-XDP programs to drop traffic at the lowest level in the driver before any computation is spent on them at all. Nearly the first thing the driver does after getting new packets in the rx ring buffer is run your program on them.
Say you have a BPF-XDP program which processes the packet to figure out what client the packet is coming from and selectively drops those packets. Is that really going to be faster than just forwarding the packet from the edge router to the next hop? I find it hard to believe that running such a program would actually alleviate full queues when all the edge router is doing is just forwarding to the next hop?
I don’t understand? The issue is that a client/customer outside of cloudflares control DOSed one of their network links. Cloudflare has no control on the client side to implement rate limiting?
I think you misunderstand the flow of traffic here. The data flow, initiated by requests coming from AWS us-east-1, was Cloudflare towards AWS, not the other way around. Cloudflare can easily control where and how their egress traffic gets to the destination (as long as there are multiple paths towards the target) as well as rate limit that traffic to sane levels.
There was definitely a recurring pattern at AWS where a single customer would trigger latent bugs/undercapacity resulting in outages. Postmortems would often recommend developing per-customer observability and mitigation.
You'd be surprised how low the capacity of a lot of internet links is. 10Gbps is common on smaller networks - let me rephrase that, a small to medium ISP might only have 10Gbps to each of most of their peering partners. Normally, traffic is distributed, going to different places, coming from different places, and each link is partially utilized. But unusual patterns can fill up one specific link.
10Gbps is old technology now and any real ISP can probably afford 40 or 100 - for hundreds of dollars per link. But they're going to deploy that on their most utilized links first, and only if their peering partner can also afford it and exchanges enough traffic to justify it. So the smallest connections are typically going to be 10. (Lower than 10 is too small to justify a point-to-point peering at all).
If you have 10Gbps fiber at home, you could congest one of these links all by yourself.
Now this is Cloudflare talking to aws-east-1, so they should have shitloads of capacity there, probably at least 8x100 or more. But considering that AWS is the kind of environment where you can spin up 800 servers for a few hours to perform a massively parallel task, it's not surprising that someone did eventually create 800Gbps of traffic to the same place, or however much they have. Actually it's surprising it doesn't happen more often. Perhaps that's because AWS charges an arm and a leg for data transfer - 800Gbps is $5-$9 per second.
Future proofing inevitable things should be something to talk about more.
For instance, people will be scraping at a "growing" rate as they figure out how everything AI works. We might as well figure out some standard seeded data packages for training that ~all sources/sectors agree to make available as public torrents to reduce this type of problem.
[I realize this ask is currently idealistic, but it's an anchor point to negotiate from.]
Thats true for several other speeds too. 100GE first generation was 10x10GbE, second generation was 4x25GbE. 200GE first version was 25GbE based and so on.
It was prolonged by the fact that Cloudflare didn't react correctly to withdrawn BGP routes to a major peer, that the secondary routes had reduced capacity due to unaddressed problems, and basic nuisance rate limiting had to be done manually.
It seems like they just build huge peering pipes and basically just hope for the best. They've maybe gotten so used to this working that they'll let degraded "secondary" links persist for much longer than they should. It's the typical "Swiss Cheese" style of failure.
Wasn’t the problem exacerbated precisely by withdrawing a BGP link because all the same traffic is then forced over a smaller number of physical links?
There's nothing to suggest the link between Cloudflare and any other AWS region has more capacity or that there aren't more disruptive Cloudflare customers using those regions.
But there is absolutely something to suggest "if you only support one region for some tasks, you're going to have problems that other people don't have."
Anyone want to tell Cloudflare that BGP advertisements at AWS are automated and their congested network directly cause BGP withdrawals as the automated system detected congestion and decreased traffic to remediate it?
It wouldn't surprise me if the BGP routes in the DCI PNI were manually configured, since this is probably one of the most direct and important connections. I would be surprised if Cloudflare didn't have firsthand knowledge of what happened with AWS during this incident.
I think the withdrawal approach by AWS would normally work, as this action should desaturate the connections. Just really unfortunate that this caused routing through a link that was at half capacity.
> The incident was a result of a surge of traffic from a single customer that overloaded Cloudflare's links with AWS us-east-1. It was a network congestion event, not an attack or a BGP hijack.
And no one knew a single thing about it until the incident. That is the current network management state of the art, let Cloudflare deal.
Braze is my guess. They let customers do a lot of stuff with pushing and pulling data per user and I would guess every customer is in a sandbox. They were also impacted by the incident.
It was the company that I found that was impacted and could have caused it. They also don't play nice with others. It's even in the documentation. We looked into buying their services. They have also caused load problems in other clouds. But it was just a guess. Could be anything.
I'm having trouble understanding the second diagram in the article. I can make sense of a directed graph, but this one has thin horizontal lines with arrows leaving them in both directions. These lines look like dividers, not nodes, so I'm not sure how to interpret it.
I think the intention is to show the divide between Amazon's and Cloudflare's responsibility, over the piece of fibre linking their network devices together. It would have been clearer to continue the lines and just put a dotted divider between them I feel.
> This system will allot network resources on a per-customer basis, creating a budget that, once exceeded, will prevent a customer's traffic from degrading the service for anyone else on the platform
How would this work practically? If a single client is overflowing the edge router queues you are kindof screwed already? Even if you dropped all packets from that client you would need to still process the packets to figure out what client they belong to before dropping the packets?
I guess you could somehow do some shuffle sharding where a single client belongs to a few IP prefixes and when that client misbehaves you withdraw those prefixes using BGP to essentially black hole the network routes for that client. If the shuffle sharding is done right only the problem client will have issues as other clients on the same prefixes will be sharded to other prefixes.
It's load shedding, but it's weighted towards people abusing their quota usually over some rolling weighted average. The benefit is that they are dropped immediately at the edge rather than holding sockets open or using compute/resources. It usually takes 30s-1m to kick in.
In this specific case, it wasn't requests from the client that caused overload. It was the responses to those requests. So Cloudflare can avoid sending responses, and prevent the problem.
You're right that this doesn't solve all cases, but it would have prevented this case.
> Even if you dropped all packets from that client you would need to still process the packets to figure out what client they belong to before dropping the packets?
In modern Linux you can write BPF-XDP programs to drop traffic at the lowest level in the driver before any computation is spent on them at all. Nearly the first thing the driver does after getting new packets in the rx ring buffer is run your program on them.
Say you have a BPF-XDP program which processes the packet to figure out what client the packet is coming from and selectively drops those packets. Is that really going to be faster than just forwarding the packet from the edge router to the next hop? I find it hard to believe that running such a program would actually alleviate full queues when all the edge router is doing is just forwarding to the next hop?
Perhaps they drop the client's flows on the host side.
I don’t understand? The issue is that a client/customer outside of cloudflares control DOSed one of their network links. Cloudflare has no control on the client side to implement rate limiting?
I think you misunderstand the flow of traffic here. The data flow, initiated by requests coming from AWS us-east-1, was Cloudflare towards AWS, not the other way around. Cloudflare can easily control where and how their egress traffic gets to the destination (as long as there are multiple paths towards the target) as well as rate limit that traffic to sane levels.
Ah I see now. Yes in that case they could just reply with 429 codes or just not reply at all.
I think you're overthinking this. Just having a per (cloudflare) customer rate limit would go a long long way.
It’s gonna turn out it was one guy on one machine calling “pnpm install” on a fast machine with a 100gbps uplink.
Can we stop with the 2015 jokes already?
I’ve actually had an npm install that failed on my ISP but succeeded with Cloudflare VPN and the OP comment was more or less the explanation.
In 2015 it would have been "npm install"
(Thanks Rauch.)
There was definitely a recurring pattern at AWS where a single customer would trigger latent bugs/undercapacity resulting in outages. Postmortems would often recommend developing per-customer observability and mitigation.
Wild that one tenant’s cache-hit traffic could tip over Cloudflare’s interconnect capacity
You'd be surprised how low the capacity of a lot of internet links is. 10Gbps is common on smaller networks - let me rephrase that, a small to medium ISP might only have 10Gbps to each of most of their peering partners. Normally, traffic is distributed, going to different places, coming from different places, and each link is partially utilized. But unusual patterns can fill up one specific link.
10Gbps is old technology now and any real ISP can probably afford 40 or 100 - for hundreds of dollars per link. But they're going to deploy that on their most utilized links first, and only if their peering partner can also afford it and exchanges enough traffic to justify it. So the smallest connections are typically going to be 10. (Lower than 10 is too small to justify a point-to-point peering at all).
If you have 10Gbps fiber at home, you could congest one of these links all by yourself.
Now this is Cloudflare talking to aws-east-1, so they should have shitloads of capacity there, probably at least 8x100 or more. But considering that AWS is the kind of environment where you can spin up 800 servers for a few hours to perform a massively parallel task, it's not surprising that someone did eventually create 800Gbps of traffic to the same place, or however much they have. Actually it's surprising it doesn't happen more often. Perhaps that's because AWS charges an arm and a leg for data transfer - 800Gbps is $5-$9 per second.
Future proofing inevitable things should be something to talk about more.
For instance, people will be scraping at a "growing" rate as they figure out how everything AI works. We might as well figure out some standard seeded data packages for training that ~all sources/sectors agree to make available as public torrents to reduce this type of problem.
[I realize this ask is currently idealistic, but it's an anchor point to negotiate from.]
Downloading cached data from Cloudflare to AWS is free to the person doing the downloading if they use Internet gateway
Hot take. 40 Gbps is not a real rate; it's just four 10 Gbps in a trenchcoat stacked on top of one another!
Thats true for several other speeds too. 100GE first generation was 10x10GbE, second generation was 4x25GbE. 200GE first version was 25GbE based and so on.
That's what started the incident.
It was prolonged by the fact that Cloudflare didn't react correctly to withdrawn BGP routes to a major peer, that the secondary routes had reduced capacity due to unaddressed problems, and basic nuisance rate limiting had to be done manually.
It seems like they just build huge peering pipes and basically just hope for the best. They've maybe gotten so used to this working that they'll let degraded "secondary" links persist for much longer than they should. It's the typical "Swiss Cheese" style of failure.
Wasn’t the problem exacerbated precisely by withdrawing a BGP link because all the same traffic is then forced over a smaller number of physical links?
AWS us-east-1 is now taking down other providers.
Only real long term mitigation is to move to another aws region; us-east-1 seems to suffer from all kinds of scaling challenges.
There's nothing to suggest the link between Cloudflare and any other AWS region has more capacity or that there aren't more disruptive Cloudflare customers using those regions.
But there is absolutely something to suggest "if you only support one region for some tasks, you're going to have problems that other people don't have."
yeah but us-east-1 is cursed
Anyone want to tell Cloudflare that BGP advertisements at AWS are automated and their congested network directly cause BGP withdrawals as the automated system detected congestion and decreased traffic to remediate it?
It wouldn't surprise me if the BGP routes in the DCI PNI were manually configured, since this is probably one of the most direct and important connections. I would be surprised if Cloudflare didn't have firsthand knowledge of what happened with AWS during this incident.
I think the withdrawal approach by AWS would normally work, as this action should desaturate the connections. Just really unfortunate that this caused routing through a link that was at half capacity.
The way I read the blog post, it seems they're very aware of that.
I imagine Cloudflare and AWS were on a Chime bridge while this all went down, they both have a lot at stake here.
> The incident was a result of a surge of traffic from a single customer that overloaded Cloudflare's links with AWS us-east-1. It was a network congestion event, not an attack or a BGP hijack.
And no one knew a single thing about it until the incident. That is the current network management state of the art, let Cloudflare deal.
I wonder which customer triggered this…
Braze is my guess. They let customers do a lot of stuff with pushing and pulling data per user and I would guess every customer is in a sandbox. They were also impacted by the incident.
It sounds like anyone relying on Cloudflare and AWS us-east-1 were impacted. Not sure it's quite the smoking gun you're implying
It was the company that I found that was impacted and could have caused it. They also don't play nice with others. It's even in the documentation. We looked into buying their services. They have also caused load problems in other clouds. But it was just a guess. Could be anything.
Also curious if it was legit, misconfigured, or attack traffic.
I'm having trouble understanding the second diagram in the article. I can make sense of a directed graph, but this one has thin horizontal lines with arrows leaving them in both directions. These lines look like dividers, not nodes, so I'm not sure how to interpret it.
I think the intention is to show the divide between Amazon's and Cloudflare's responsibility, over the piece of fibre linking their network devices together. It would have been clearer to continue the lines and just put a dotted divider between them I feel.
There are some missing y-axis labels that would be interesting to see
Come on, some CF insider please create a throwaway account and tell us which client was :)
[flagged]
Didn't even notice