Maybe it's just the tags I follow, but web-scraping and tools that facilitate it seem to be very popular for questions these days. One category of question I see frequently is some form of:
I'm trying to scrape (some website) and my code was working fine for a while but now I'm getting (a 401/404/429 or some other HTTP error | an empty response | a timeout). Please help!
Nearly always it is the server and not the client that is responsible for this change in behavior, and nearly always the issue is that the server administrator doesn't want this kind of scraping and is trying to prevent it, using some kind of web application firewall or other bot-detecting mechanism, and serving errors or tarpitting the connection intentionally.
There are various techniques to evade server-side detection or to circumvent limitations, but these techniques are mostly not language-specific or tied to a particular website. This argues in favor of a language-agnostic question that can collect high-quality canonical answers about techniques rather than about implementations of those techniques in different languages. (For example, various kinds of back-off and request rate limiting, using a CSRF token from one request to build the subsequent request, etc.)
More importantly, I feel like such a question could also be a good place for a canonical answer explaining the importance of looking for alternatives to web-scraping, such as using a documented API when available. Many web-scraping questions are ultimately aimed at subverting the terms and conditions of the sites in question, and I feel like that should be mentioned as well, since there are legal and ethical implications of that. (I'm not saying that Stack Overflow should censor questions about User-Agent randomization, but just that we should point out that if you randomize your User-Agent with the sole intention of circumventing a rate-limiting mechanism, you're pretty clearly operating contrary to professional ethics.)
(On the flip side, there might also be a place for the "opposite" canonical question, about how to detect HTTP requests from web-scrapers or other bots and how to respond to them. Answers could describe things like request fingerprinting, rate-limiting and quota, and so forth. I say "might" in this case because relatively few people are actually implementing such things—most websites that are big enough to care will already rely on Cloudflare or Akamai or some other edge network to handle this stuff for them.)
In any case, I'm not aware of any existing canonical question. Is there one anyone can recommend? If not, I can create one, but I figured it would be better to ask the community first.