We recently were hit by multiple DDoS attacks over a weekend. We have our own servers in a data center with 5 redundant 1 Gbps links. The DDoS was 20Gbps according to the upstream providers.
Our upstream implemented layer 7 mitigation which did an unbelievably effective job at stopping the attack in it's tracks. I don't know the tech that they used, but it performs deep packet inspection up to the application layer and they charge a modest additional fee for passing our traffic through that system.
The effect was that our traffic dropped to very slightly below normal levels during the attack, which would indicate that there were probably a few false positives, but we didn't have a single customer complaint.
Thanks for taking this stance. I'm a customer (I think we actually talked together recently about Apple/Google vCard implementation) and I appreciate you guys making a stand against this criminal behaviour.
It could seem to some people that such a stance is easy, but no matter the strength of principle, when you see your business go offline and customers start banging at the doors for you to sort it out then the situation becomes more complex.
So from this customer - keep up this stance, keep being transparent and I for one will stick by you guys without hesitation!
Right from the start, I don't have a good feeling about protonmail. On their About Us page, the CTO self-proclaimed himself a crypto expert. The CTO has a phd in particle physics and has never published anything on computer security.
Turns out they are the type of people who pay to make the problem disappear.
DDoS attacks today are such a commodity. It takes next to nothing to launch them.
You can get upwards of 200Gbps for 1/6 of a bitcoin. It is very easy to setup and you can DDoS your favourite site in a matter of minutes.
These are not very smart attacks and can be mitigated even using the free tier of cloudflare.
I don't have the background on the mail provider attacks but 6.5k ransom seems to come from attackers who use easily available booters. Hushmail switched to Cloudflare throughout their attacks and that seemed to have helped, not sure what fastmail will do.
But any public web service should not be in a position where they are vulnerable to off the shelf DDoS attacks.
DDoS attack as suffered by for example Githuh with heavy coordination and nation states behind them require more specialised defenses. There are commercial alternatives out there that go from anywhere between 9k-40k per month depending on bandwidth and technology - see Imperva, Prolexic, Neustar, Nexusguard, Blacklotus, Incapsula, etc..
Apart from the initial setup which is more involved than Cloudflare's there is not much to do apart from throwing money at it. Quite the money making business really :)
If we were pure web we would have ducked behind Cloudflare immediately. Since we do SMTP/IMAP/POP3 as well, we've had to go with a more complex (and costly) solution.
This is our theory for why they're currently attacking email providers. We're not "just a web site", and attackers realise that the situation is more complex for email sites, we can't just hide behind Cloudflare.
We're not sure who's actually attacking us, the ransom note comes from a freemail provider and a connection from a tor exit node. We can only guess at their total capabilities.
Sorry to hear about that... Do get in touch with one of the providers listed above, they may be able to help you out in the short term for free in exchange for publicity.
0. Place an HA pfSense CARP or OpenBSD pf CARP setup as a pair of transparent proxies in front of everything (eg at the edge on the other side of HA network gear with either 2 (or 3, if deploying a private, admin network too) NIC teams for isolating traffic). This will let you do raw L3 traffic measurements on each side with graphite/collectd, cacti, rrdtool, etc. and L2/L3 IP/network banning (if you don't own/admin the network gear or don't want to touch it in production). These are super cheap and only need ~128 MiB RAM each and very little CPU and disk (except for logging, you want a dedicated PCIe SSD or SSD partition if possible). (Your public IP(s) should point to these boxen.)
1. Definitely get stuff behind reverse SMTP/IMAP/POP3 proxy like nginx or haproxy.
3. There are many other tweaks and there are some appliancized VMs for anti-spam and DDoS that can be dropped behind the trusted network-side. (I would advise against Cloudflare-like services for most mature and non-web apps because they are add'l points of failure and increase latency, and they duplicate what good sys/netadmins implement routinely, especially if you're already deployed to multiple DCs servicing multiple continents and/or geodns.)
Pedigree: I'm a founder and once-upon-a-time security researcher & sysadmin whom sold out and became SRE manager and then a consultant. I used to maintain multiple deployments of commercial Zimbra (from m&a activities) for clients including hi-ed, non-profits, VIP individuals, and enterprises.
That doesn't fix the actual source of the problem; if traffic ingress costs you money then they're literally burning your money via that attack of attrition.
The way that most of the 'protection' sites work is that they host so much aggregate traffic that it still totals out to more than the incoming attacks/normal traffic and they can literally just eat it at almost no cost (their service is doing that, and some form of filtering to keep it from reaching the actual target; the 'extra' cost for the attack is almost nothing since the processing hardware is nearly fixed in cost).
The only way to handle this from a peer to peer perspective is to be able to send the electronic equivalent of 'gag orders' at hosts/ranges that are misbehaving (and have them stick, either by the other edge or by upstream providers there of). Said orders wouldn't be enough, alone, to warrant quarantine from the Internet, however a number of different sources indicating infected behavior would be.
This doesn't stop traffic from getting to you. If the attacker has more bandwidth than you, you lose. Complete outage. All your customers think you're down because they can't get packets through.
You need the DDoS traffic filtered upstream, not stopped at your firewall. This is not a trivial problem to solve.
If you're running BGP you can stop small-ish DoS attacks by setting up a blackhole BGP community that is propagated up to your providers. Any IPs you put in will no longer have their traffic forwarded. This doesn't work for DDOS which has countless IPs attacking you. You will have to blackhole yourself and take the target IP off the internet to stop wasting all your bandwidth.
So the solution is to have it filtered upstream by someone who can clean and absorb the attack. It adds complexity to your network architecture (filtering provider has to announce your routes for you) and it's not cheap.
this is the first I've heard of being able to block traffic in excess of 10Gbps with "128MiB" and "very little CPU" in addition to Cloudflare-like services adding latency.
Yeah, it's small-time advice. Good advice for protection against complexity attacks, not so much for protection against tens of Gbps of random junk that fills your entire pipe.
(we do run nginx on out frontend machines for both web and mail protocols, protecting the Cyrus servers behind it from complexity attacks and providing fan-out connection routing)
We dropped all the DDoS packets at our edge firewall quite comfortably - users wouldn't have even noticed except that it filled up our incoming links, so packets started dropping.
I'm really quite impressed at the tech which the big DDoS protection providers have for packet inspection and cleaning the feed before it reaches the end host.
It does lower the overall egalitarianism of the internet to have to deploy defenses - we lower our overall routability to put these mega filters in front of incoming packets - but that's the reality of a world where fiends can control tens of thousands of boxes and have them spew traffic at any random network address. You need to filter out at the boundary.
Nothing short of filtering beforehand can stop a channel from being filled if it gets more than its capacity per second of incoming packets.
The only real defense is to serve your site from somewhere with more incoming bandwidth than incoming abuse. Afaik, the only type of filtering you're likely to get from an upstream provider is null routing of attacked ips, which helps protect their network, but doesn't help you serve users (you can switch ips, but abusers will likely switch too)
We used to use TCP proxies when I ran a game server where skiddies would try to attack it. I know buyvm.net is where we got one small $15 a year VPS with $3 a month "DDOS protection", not sure if we ever had issues after that with DDoS or not, we kept getting more and more proxies. We would give users a different one upon refreshing the page (it was a browser based game, but they would connect to a game server via TCP). I know the Minecraft community offers proxies as well. If you're on HTTP though, cloudflare is one of the cheaper options.
I've just started checking out Fastmail as a result of this thread. I hadn't heard of it before but I'm really into it.
I've been trying out different email services lately in an effort to untether myself from Google, and this one seems like a really good choice.
I'm not into the iOS themed non native Android app though, but being an Android developer I'm more averse to that nonsense. It seems to work great and is really fast though. Does anyone know if there is a native app planned?
I'd be willing to pay a subscription for email if it's worth it. I'd love to hear others' experience with them.
> I'd be willing to pay a subscription for email if it's worth it. I'd love to hear others' experience with them.
It's a very good no-nonsense service based on standard protocols, although a bit pricey (compared to a grandfathered, free Google Apps account). They've lately been expanding to offer a more complete offering (CalDAV, CardDAV, etc).
This means it plays nicely with any platform where these standards can be applied (Googley or not) and it also means it's easy to take data ownership seriously, if that's your thing.
I only have positive experiences with fastmail, and I like that they don't have messed up and bloated their WebUI like Google have.
Currently I'm only using it for my personal email because of price. It's not that expensive, but because it's 100% free for me, I still have the rest of my family on the Google apps account.
The reassuring thing about paying for a service like this is that you know who the customer is. Fastmail will not change their UI to sneak in a "social network" in your inbox just to mine more data and milk more ad-dollars out of you.
I have a free GApps account, but I ditched it in favour of FastMail - primarily to extract at least some of my life from the Google juggernaut, and also to support the little Aussie battlers. Thanks for offering an awesome, rock solid service, and supporting Open Source so well.
I'm seriously looking forward to JMAP, I hope some other big services adopt it when it stabilizes. Give your JMAP dev(s) a hug from me, please!
Oh, and I hope the DDOS doesn't cost too much - any chance we could have a blog post about the costs of the DDOS attack after it's all over? I know most companies don't like talking about operational costs, and I'll understand if you guys are the same. Cheers!
We're definitely planning another post at some point about the systems that we've developed over the past couple of days and the additional services we've purchased to help us ride out the storm.
I'm looking forward to JMAP as well - there are a few of us working on it (I wrote the proxy, which is in serious need of some love)
I have not shopped around for mail, so I have nothing to compare it with. But their tiers are $10/$20/$40 for .25/1/15GB per year. To me that's a _tiny_ price. I don't know what you mean by "pricey" then.
It's not pricey, people are just not used to pay for email.
I don't remember the exact numbers, but it's more or less the same as Google with a custom domain.
BTW, Fastmail is amazing, been using it for a while now and super happy. It's solid and has some really good features.
> But their tiers are $10/$20/$40 for .25/1/15GB per year.
Which means if I were to migrate all my (perma-free) Google Apps account (for the whole family) and were to retain my current quotas, I would have to shell out $40*10 or so.
Fastmail is nice and all, but I know lots of ways I'd rather spend $400 which doesn't involve being tech support for 10x family members needing help having their accounts migrated on all their devices.
It's not fair to call Fastmail "a bit pricey" when your point of comparison is a special circumstance you are in that is not available for new users. In general, Google Apps is $50/year, which is 25% more than the comparable Fastmail plan.
But it's not really comparable since Google Apps has a much better storage story (Google Drive vs. WebDAV-based storage) and an office suite with collaborative editing, etc.
Also, additional storage is much cheaper. You get 100GB extra storage for $1.99 per month or 1TB for $4 per month (or 'unlimited' if the domain has more than five users).
But I agree that $40 per year is not pricy for a fast e-mail service with a lot of redundancy. Plus, Fastmail contributes a lot to open source projects such as Cyrus.
(Note: I have both a Google Apps and a Fastmail account.)
> Google Apps has a much better storage story (Google Drive vs. WebDAV-based storage)
I've been a Dropbox user even before switching off Google Apps, because Dropbox has a much better client and compared to alternatives, storage for them is not just a complementary to something else, so for example they support Linux as well. Which is very important for a multi-platform guy such as myself. Dropbox is also integrated with Microsoft's Office Online, with Gmail (by means of a Chrome extension) and with Fastmail's web interface. So from Fastmail's web interface you can attach files straight from Dropbox. Plenty of apps have integration with Dropbox actually, like for example 1Password. Dropbox is also the suggested alternative to iCloud by Apple.
And that's not the only option. If you're a power user interested in security, there's also SpiderOak. It's a bit more pricey, but that's because they are doing encryption and so cannot take advantage of duplicate files and other gimmicks like that. And it's worth it for people worrying about the privacy of their data.
> 1TB for $4 per month
The pricing you're talking about is about the Vault option and is $5 per user per month and not $4. And the big problem is that's misleading. That's $5 per month per user and is applied for all users in your Google Apps account, whether they need it or not. My wife for example certainly does not need 1 TB and for small businesses that can be very problematic, as you can easily pay an extra $100 per month.
In other words, I see no reason to encourage a monoculture on the basis of tighter integration or complementary pricing that's misleading. We've been experiencing this strategy time and time again in the past from companies like Microsoft. You'd think we should have learned by now. In fact such marketing strategies are exemplified in books such as "Predictably Irrational" by Dan Ariely. That was an interesting read if you're interested.
And it never ends up well, either for consumers or for the industry at large. And you've got good options available that I think are better than Google Drive.
> an office suite with collaborative editing
But nobody stops you from continuing to use Google's Docs, in combination with Fastmail and Dropbox or whatever. I've done that, it's not bad and should not be a reason to keep using Gmail. Big companies like Google, Microsoft or Apple want you to get from them everything but the kitchen sink, because that's how they achieve lock-in, that's how they can use their brand muscle to make you buy shit you don't need or stick to inferior options. You shouldn't forget that Gmail is about email and if Gmail no longer does email well for you, then complementaries like Google Drive or Google Docs won't make it magically work better at email.
But btw, did you know that Microsoft's Office Online can edit and save files as ODF, the standard document format and Google Docs does not support ODF? In fact Google Docs doesn't support editing any of the common formats, as they require conversion in their own format in order to edit those documents, leading to a form of lock-in that Microsoft has only dreamt of.
> Dropbox ... so for example they support Linux as well
Apart from the idea of not putting all my eggs in one basket, this is crucial for me too.
Dropbox is the only cloud-storage service with a good and working Linux client. All my machines at home runs Linux, so not supporting that means I wont even consider using the service.
I've been a Dropbox user even before switching off Google Apps, because Dropbox has a much better client and compared to alternatives,
Oh, definitely. Outside mobile, the Dropbox client is miles ahead. I was just saying that a Google Apps account and Fastmail account is not directly comparable, since Google Apps offers so much more.
If you're a power user interested in security, there's also SpiderOak.
I don't see the added benefit. As long as the standard client is closed source, it's only a bit better from a security perspective.
That's $5 per month per user and is applied for all users in your Google Apps account, whether they need it or not.
Definitely. But we were comparing to Fastmail, where storage costs 1GB for $5 USD/year for enhanced accounts. Just for comparison, for 100GB that is $41 per month above the base account cost, compared to $1.99 per GMail. Then a $4 per month account plus $1.99 for 100GB or $5 for 1TB doesn't look so bad.
But nobody stops you from continuing to use Google's Docs, in combination with Fastmail and Dropbox or whatever.
If you use Google Docs outside Google Apps, your documents can be mined for advertising. No thanks!
You shouldn't forget that Gmail is about email and if Gmail no longer does email well for you, then complementaries like Google Drive or Google Docs won't make it magically work better at email.
I have Fastmail and Google Apps and I still like Google Mail more in general. For instance, I prefer labeling over folders and the mobile GMail/Inbox apps are a far better experience than the Fastmail app.
But I don't agree with the premise. One of the nice things of Google Apps is integration, e.g. mail <-> calendar, Google Now, and Inbox. Dropbox realized how important this is and started pushing integration beyond providing an API for apps (Office Web integration, Office plugins, Google Mail extension, etc.)
> I prefer labeling over folders and the mobile GMail/Inbox apps are a far better experience than the Fastmail app.
I use Fastmail, and they support standard internet protocols like IMAP and CardDAV. Which means I can use the standard Android email-apps, CardDAV sync, and have everything still work just fine.
No need to use an email-provider specific app, although Google and Gmail has tried to mentally brainwash everyone and their grandmothers that this is how email actually works.
However it's a total non-starter if they can't archive email forever. Bonus, like faxing still is for some in business, today's worker (and end user) is going to force everything through that hole, because it's the magic service that just makes it work... and when it doesn't they can blame it.
As mentioned, OP had a grandfathered Google Apps for Business (so, free forever?). Another alternative is to sign up to a $5-$10/month web hosting package which normally comes with free email hosting.
Disclaimer: I'm a FastMail customer, been really happy with their service :) I also have a grandfathered Google Apps for Business account.
FWIW I'm just switching over from using a web hosting package for my email towards FastMail. I was generally happy enough with the web hosting package, but didn't actually need it for anything much other than the email in the end. The biggest benefits I've noticed are reduced spam and a web interface that works well - I can actually search all of my email from my phone now, which was previously a practical impossibility :-).
I hadn't noticed any slowdown, so kudos to the Fastmail guys!
I've used them for six years on a paid (enhanced) account and they've been very reliable for me. I've never noticed any downtimes and I appreciate how fast their web portal is when I need to mess with my configuration. There's also been zero issues with CalDav.
I'm sure other providers give similar access, but they also make it very easy to create alternative email addresses for the domain I own. Once I started using them, I started setting up a different email address for every company I did business with online and that's cut down the amount of spam I receive dramatically. Basically, when I see a compromised email address, I disable it and that seems to do the trick.
In any case, they've provided me rock solid service and that's worth the $40/year that I pay.
Great experience with our business account. We pay some trivial amount and our emails are out the door... fast. There's also the basic support that you would expect built in. Our experience is that they just "do one thing well". It's probably the second-to-last external dependency we'd [edit: internalize], with Stripe being the last and Analytics and Github being the first two, if that gives any perspective.
I've been using it for my main domain for over a year... It's a great service and I'm happy with it. I wanted to move away from GMail, without losing a good spam filtering solution, and it works very well. I recommend Fastmail.
I pay yearly for them to host email for my personal domain. No problem with email delivery (so far) and now that they let you flag emails as spam from the notification in the android app, I'd say that they do everything I want.
Ideally of course I'd be self-hosting, but that's a tarpit I'll leap into at some time in the future.
I have a lifetime email alias from my university, and when I used it with Gmail it would always appear to people as "From me@gmail.com on behalf of me@wherever.edu". I wanted my .edu to be my actual email address, but my gmail address is what was ending up in people's address books.
So I tried out Fastmail with my alias and had no trouble getting it working the way I wanted it to work. So I've been with Fastmail ever since.
Moreover, you don't want to use a different 'from' address without sending through the appropriate SMTP server. If the domain has SPF or DKIM set up, a receiving server might reject your mail if it wasn't sent from an expected SMTP server.
I have my GMail configured to send email from my .edu address through my school's SMTP server. Fastmail supports this too. Look at your IT department's website, they probably explain how to configure it.
Another +1 for Fastmail. I've been putting off migration from Google for months, but finally got around to moving things over. The migration was easy, their UI is shockingly quick (about halfway through processing ~20k archived messages), and it's got most of the features you could possibly want.
+1 for Fastmail over my 6-months experience: Reliable service, fast UI, no G+/GNow/Promotions tab annoying push.
However there are things I'm really used to in Gmail, for example the unread messages at the top, and seeing messages in several labels. So Fastmail feel a bit less advanced.
Labels is something that's tricky to implement on top of our IMAP-standards-folders-based server, but we have some ideas on how to do it. Sort unread to the top is easy on desktop interface, though there's no UI to do it on mobile.
search is:unseen works nicely to find them on mobile as well, and you can save it to your sidebar/folders list.
On iOS, the client seems like it is just a shortcut to a website. However, I use IMAP in other applications, whether Apple's Mail or Outlook, and I'm just as happy if they don't develop a complete client.
I've been using Fastmail for 2 or 3 years, recently moved my XMPP and Caldav stuff over too. I would recommend them, no complaints, I've not noticed a single outage in that time.
It was trivial to IMAP import mail from my old host. Good support for multiple authentication schemes, i.e. I have a 32 random char master password saved in my password manager, and they allow me to add any number of alternative authentications like shorter passwords that are only allowed in combination with Yubikey/Google Authenticator or one time passwords.
As far as I can tell they offer the same services as https://kolabnow.com/ at a comparable price. The difference I can see is that kolabnow uses Open Source software throughout and contributes back. That's why I chose to support them instead.
On my third year of membership and wouldn't hesitate to recommend Fastmail.
Only ever had one issue (a small bug to do with some very specific domain config) - their support team emailed me back straight away, it was escalated to the devs, and fixed almost immediately.
Now with CalDAV and CardDAV support it's a no-brainer.
Overwhelmingly positive, with great support too. I've seen their devs commenting on GitHub issue threads tracking down sync issues - deeply clueful and prompt support. Plus all the things: family accounts, CalDAV, CardDAV, XMPP...
I moved to Fastmail six months ago based on their reputation. I use it for several accounts, including that sent to my own domain name. No regrets at all.
Given that so many of us are now hosting on AWS, I'd like to ask the question - who has been hit with a DDOS attack / extortion letter while is hosting on AWS? It would seem that there's many old-tech companies hosting in data centers that would seem to be far more vulnerable to non-TCP attack vectors than AWS-hosted systems. Is that who is generally targeted here? Are there any stories, anecdotal or otherwise, about people getting hit with DDOS attacks while using AWS. Here's a talk by AWS on their measures against attacks - https://www.youtube.com/watch?v=Ys0gG1koqJA. The only thing short of Silverline etc defense that they seem to be lacking is the reporting dashboard indicating when they've defended against DDOS attacks. So has anyone received a letter from DD4BC and other miscreants whilst hosting their domains on AWS?
It's nice to see this published proactively. At the very least, transparency like this helps users understand what the circumstances are ahead of time.
Best of luck to the Fastmail team, I hope they are able to weather the storm out.
And they've very clearly refused to pay the Danegeld demanded by their attackers. Smart, principled and yet another reason I'm a happy customer of theirs.
I have not noticed any service interruption at all. Well done in handling this attack! This is just one more reason why I will remain a loyal FastMail customer.
Wow, seriously, fuck these people. There has to be a technical solution to this, since it's infeasible to find/fix/finish the Armada Collective.
SMTP/IMAP/etc. are pretty crappy protocols in a lot of ways, but they're what everyone has deployed. They can be proxied like HTTP/HTTPS. There are spam/reputation issues with outgoing traffic, too, which makes this even more annoying.
We're a FastMail reseller, and so far we haven't experienced any problems - either on delivery, or the front end interface. So good job defending against it.
What I love about FastMail is that I can neatly organize my emails into folders based by categories (via rules for subdomains), such as: newsletters folder (service_name@newsletters.mydomain.com), social (service_name@social.mydomain.com) etc. Everything is so clean and I don't even see any spam, they're top notch.
DDoS is a type of attack that is so old that I wonder why it is still possible to exploit it. It is at least 15 years old? (edit: I am not blaming FastMail, this is just a general assumption)
It's still possible to exploit because "DDoS" isn't one attack, but a category: people keep finding new ways to slam a target with tons of traffic.
For a while, the most common way was with botnets of compromised PCs. They still exist, but big attacks with them are less common since Microsoft has gotten better at securing people's computers. The big thing now is "amplification attacks": basically, finding a way to send a small amount of data and get some other host to flood your target with a huge amount of data in response. Search "NTP amplification attack" for details. More recently, China has weaponized the Great Firewall to be yet another DDoS vector: they inject JavaScript into pages that people visit, and that JS floods a target with requests.
As long as there is some way to point a lot of requests somewhere you want, DDoS attacks will be a thing.
Most attacks are trivial to detect, you don't need AI. It's just hard to get useful work done when all your incoming interfaces are overloaded with easily detectable abuse.
The "best" way would be to have application logic to detect non-legitimate requests, and make an API call out of band to upstream networking gear to insert a null route for that IP (so as to drop the traffic at the edge before any real "work" takes place on it).
In a previous life, I ran physical datacenters, and while the gear wasn't terribly powerful then (we're still worried about running out of memory on core routers, hence why IP blocks don't get sliced up and piecemealed out with the exhaustion of IPv4 space), I'd expect newer hardware to be able to keep up.
The network can remain irrational longer than you can stay online.
Problem is, beyond a certain volume, even the upstream gear is gonna get saturated just reading the header on the bogus packets and directing them into the bit bucket. It's not unheard of for the larger attacks to take down entire ISPs.
Unless you keep propagating "upstream" and the message gets to the "source" ISP, and they block the actual misbehaving user/account. For all we know, they can kick them off the network after sufficient transgression, and ban their account at the hardware ADSL level (assuming that's what it is). This also presumes the ISP is willing to implement such a feature, and kick-off their paying (albeit infected) users.
I don't know much about this stuff, so I'm extrapolating and pseudo-solving.
Usually there's a lot of diversity in the immediate source of the traffic. If it's a volumetric attack, the immediate source is the misconfigured servers that spoofed packets are being sent to. If it's an in band attack, the immediate source is usually botnet members, but occasionally regular browsers being served bad scripts by a compromised or mitmed site.
You could work to notify the network owners, but it's whack-a-mole; even with strong efforts there are enough DNS and ntp servers out there configured to generate a pretty big reflection.
It's a generic term, not a type of attack. The question is similar to asking why we haven't developed a defense against being blown up when explosions are old.
As long as there are services, and as long as those services have finite capacity, there will be DDoSes -- both accidental and intentional.
Several people seem to be saying that DDoS isn't one attack, which is correct, but not particularly relevant. Many times the attacks are simpler today, because there's little need for sophistication. Previously many attacks used to use some form of multiplier, either for the traffic itself or the resources targeted.
The reason the attacks are still viable is because little has happened to the Internet itself. We still have the same challenges we had 15 years ago. Some things are slowly getting better, but it's still fundamentally the same. Increase in overall bandwidth and vulnerabilities doesn't help either.
We need technical capability for all the tier 1/2 boys to be able to tell if packets incoming from network X are legitimate, and drop them if they are part of currently ongoing attack.
Afaik fixed silicon edge routers peering tier 1/2 networks were the biggest obstacle in filtering good traffic from spoofed/botnet one. Just a year ago we had huge problems when BGP rose to >512K entries, which is an order of magnitude easier.
Did anything change since ~10 years ago? Last time I dabbled in this it was so bad even Tier3 (ISPs) werent filtering spoofed packets.
Not even more servers - fatter pipe. We are using less than 1% of the inbound capacity of our connection in normal operation, yet the attackers easily filled it.
The firewalls that we are running were easily able to absorb the additional traffic. Neil's metaphor of the post office was quite accurate - even in the heaviest attack, when we didn't have upstream DDoS protection turned on, about 10% of user traffic was getting through just fine - it's just that a random 10% of traffic makes for a very poor TCP experience.
There's too many ways to do it by abusing legit protocols, tricking web browsers, etc, what fool proof countermeasure would you think there is? Other than out-doing the attacker by over provisioning your networks?
Ddos traffic generation capacity and server capacity scale in similar ways. As more and more of my servers get 10g Ethernet, more and more idiots are running chargen on 10g Ethernet.
At Google's scale, all connections are symmetrical, and they send a lot of traffic than they receive. So they have a huge amount of incoming bandwidth available to absorb a volumetric attack. They also have a skilled traffic engineering team who can adjust routing or implement filtering at the edge of their network. And a skilled security team who can react to trickier attacks (bogus, but well formed requests).
Speaking of routing, I don't know for sure but I'd imagine most if not all of Google's services are on anycast IPs - meaning each node taking part in the DDoS would have its attack routed to its nearest Google DC rather than any central target.
On a global scale this would mean that the one big DDoS you'd expect to see effectively gets split into many tiny DDoSes, which Google can handle using methods mentioned in the other responses in this thread.
One would assume DDoSing the biggest service around, run by a company that has experience running giant internet facing services, a lot of security experience and tons of cash would be a lot harder than a provider with in comparison small connectivity and a small number of locations, yes. Pure size helps, + if you are at the scale of Google your upstreams probably have more interest in helping you if it were necessary.
It would be interesting to see how often people try though.
It may even been simpler than that and they might not even need 'upstream' any more. They may not even have upstream for many things, given their scale now, they probably just peer to Tier 1, if they even need that any more! (they were actually just as big as Tier 1 folks back in 2010![0].
Google has been buying up 'dark fiber' for years and has thousands of miles of cable connecting their data centers.
They can certainly handle petabit/s levels of traffic inside the datacenter[1], it's not that much of a stretch to think that they can handle double digit terabit/s through their collective external fiber links.
Also, just think about their normal level of operation. Even just all the Android devices feeding data back and forth, let alone analytics, maps, gmail, search etc etc. They've got 36 data centers and co-locate in more than 60 public exchanges (and that was in 2010!), not to mention the Google Global Cache (GGC) servers inside consumer networks across the globe.
Their scale is ridiculously large. I suspect that they actually can't be DDoS'd in the normal 'chuck traffic at them' sense.
Yeah, when you run websites that everyone accross the world hits at any given second all at once, I think a DDoS might tickle it a bit, but it would take a pretty massive botnet possibly, maybe a nice chunk of legitimate Google visitors to affect Google?
You could say that Google is constantly under a DDoS attack because the amount of normal traffic they get far exceeds the amount that would constitute an attack for an average business site.
It's mostly a matter of scale. Google are big enough and their internet connections are big enough that they can absorb a regular-sized DDoS without breaking a sweat.
Back in the day (/shakes fist at cloud) it was the slashdot effect. Rapid spikes in popularity overloading systems. We certainly couldn't handle scaling up to gmail size all at once. Maybe over a couple of years.
Of course our growth strategy is quality (people willing to pay for a good service quid-pro-quo) over quantity (free service and monetise later via ads/analytics), so we've had a slow steady growth for the entire 15 years we've been operating rather than the viral growth and sell-out/pivot that unicorns are known for.
People like Ford and Edison used to believe that you could get more quality as volume increased. And in fact, if you wanted to increase quality, then you had to increase volume.
There's an "average size of humans has increased" joke in here somewhere - both height and girth.
And I do agree to a point. We're very happy to have increasing numbers of users so we can afford to do things like contracting the excellent developer who's working on JMAP support for Calendars in Cyrus IMAP at the moment, as well as hiring people to add new features or improve existing ones.
We do try to stay at a point where we can run comfortably on 50% of our hardware, so we can shut down half our machines at once for maintenance. Redundancy certainly helps - we've blogged a few times about how good it is to be able to shut down any one machine with only a few minutes' warning to move active users off it.
I'm certainly looking forward to spending time again on what I wanted to be doing (Cyrus IMAPd improvements at the moment) rather than battling a DDoS!
Indeed, Gmail maintains extensive protections against peers being DoS'd by Gmail. The number of connections and the rate of mail is closely controlled, because a lot of sites can be taken down just by opening a few TCP connections at once (what are they using, inetd on SunOS?).
Our upstream implemented layer 7 mitigation which did an unbelievably effective job at stopping the attack in it's tracks. I don't know the tech that they used, but it performs deep packet inspection up to the application layer and they charge a modest additional fee for passing our traffic through that system.
The effect was that our traffic dropped to very slightly below normal levels during the attack, which would indicate that there were probably a few false positives, but we didn't have a single customer complaint.