There Is Basically No Fundamental Way to Defend Public Web Services Against Attackers

Hello, I’m useless.

Attacks have increased quite a bit lately, so I’ll try writing down some thoughts off the top of my head.

There Is No Fundamental Defense Against Attacks from HTTP Clients

To begin with, let’s consider each type of defense.

IP Addresses

IPv6 addresses would be fine, but in a CGNAT environment, users are assigned shared IPv4 addresses, which means people other than the attack target can also get caught up in the impact.
Also, rotating IPs is easy. DDoS attacks launched using thousands of IP addresses from cheap US VPS providers—which have a large global pool of IPs and are often used for attacks these days—or Chinese cloud environments have become commonplace. Even if you block by AS number, blocking Chinese cloud IP addresses is not a fundamental solution if you have services operating in China, and it would also be a bit unfair to Chinese users.
Chinese cloud environments are very often abused by people in other countries, so I don’t think they’re something you can simply block across the board.

My basic view is that if a service is part of the World Wide Web, then as long as even one person enjoys using it, I don’t want to block it indiscriminately.
I watch videos on BiliBili sometimes, and I use US services too.

User-Agent

Basically, this is out of the question—or rather, I think evaluating request headers as a form of access control is almost entirely pointless, so there isn’t much to say.
Since User-Agents can be spoofed as much as you like, you can’t block requests based only on a simple evaluation of the UA.
One option would be to completely block requests that claim to be googlebot but whose reverse DNS hostname is not *.googlebot.com or *.google.com, as I do on this site. But you’d need to implement that yourself, or check whether something similar is implemented on the CDN side. Also, although Google officially says this, there are actually some legitimate bots sent by Google from hosts outside those domains, as determined by reverse DNS, so this check alone isn’t a perfect solution.

By the way, even a lousy site like this gets plenty of visitors claiming to be Googlebot:
fail2ban report - fake-googlebot

Blocking Countries

This could be a temporary measure in response to an attack, but it doesn’t work as a permanent solution. The reason is the same as for blocking by IP address, so I’ll skip the details.
However, it may be useful if you operate a service exclusively for users in your own country.

TLS Fingerprint

TLS fingerprints can also be spoofed easily.
The reason is that an HTTP client can incorporate the same cryptographic library used to build a browser, allowing it to spoof the browser’s fingerprint.
It’s not quite right to say this is completely ineffective, but for example, if a legitimate bot makes requests using the same cryptographic library, it could be blocked by mistake.
Since there’s a high risk of overlap with many HTTP clients, you can’t evaluate requests based on this alone.
So it can be useful as one factor in a decision. Rather than using it as the sole condition, it may have some value when combined with other rules.

Cloudlfare Challenge / PoW Challenge

Cloudflare Challenge itself can be bypassed easily without even using a headless browser.
I’m not sure about Anubis, but it doesn’t offer a great UX, and it requires client-side computation, which can be pretty frustrating when navigating between pages.
In a sense, it can provide combined protection alongside a JS Challenge. In fact, more and more leak forums these days are implementing their own PoW Challenge. So, where should you use a PoW Challenge? If you’re using it to protect against DoS, it’s debatable whether to protect at this app-layer-adjacent level. If you implement it only in carefully chosen scenarios, you might be able to build a relatively defensive front-end implementation while minimizing the impact on UX.

Pay Wall

I’ll skip this, since only a limited number of services would want to use a paywall.

HTTP Caching

I wasn’t sure whether this belonged in the article at all, but I included it.
CDN caching can help prevent requests from reaching the origin, for things like GET requests, but my impression is that pretty much everyone has already implemented it.
Rather than thinking of it as protection, I think it’s better to use it to improve the user experience and, conversely, to mitigate floods when returning error responses—in other words, as a way to improve performance without affecting the application.

Rate Limiting

The key itself varies quite a bit depending on the service, so it’s probably best to check the implementation rules carefully. For Nginx, for example, you can apply it to overall traffic, individual URIs, individual IPs, or a combination, so I think it can be useful as a DoS mitigation measure.

So, What Should We Do?

I think the only option is to divide things across several layers and respond using a combination of conditions.
What I’m considering at the moment is allowlisting three reliable cloud providers by AS number, along with crawlers from various social media companies, and being a bit stricter with everything else.
Some request patterns are clearly suspicious and typical of attackers:

• IPs that scan for .env and similar files

• Obvious fakes claiming to be googlebot

• Sending impossible requests via POST/PUT and similar methods

• Sending all kinds of nonsense requests to paths such as /login or other endpoints that can create sessions

For things like these, maybe blocking them outright is fine? To be honest, cheap VPS providers are abused a lot, so part of me wants to block them all, but I often run a VPN server on a cheap VPS myself and browse sites using its exit IP, so that might be overdoing it. Depending on the service, though, it could make sense.

Also, audio-device detection-based fingerprints have been criticized lately, but I do think they’re actually useful, so it may be reasonable to block users when detection fails. They’ll probably find ways to spoof it eventually, though...
Maybe in the future we’ll see GPU Fingerprint, Audio Fingerprint, and other techniques adopted together to identify users’ devices.