How Websites Detect Bots in 2026 (and How Scrapers Stay Compliant)

Two clients ask the same site for the same page in the same second. One gets the data and moves on. The other gets a captcha, then a soft block, then nothing. The requests looked almost identical, but the decision was made before either connection finished setting up. That gap is where all of bot detection lives, and understanding it is the difference between collection that runs for months and collection that dies on day one.

I run proxy infrastructure and production scrapers, so I have spent a lot of time on both sides of this wall. This is a defensive explainer, written from the site’s point of view: how modern detection actually works, and how a scraper that respects the rules stays clean. It is not a guide to evading anything, because that version does not last and it is not the business I am in.

What detection is actually looking for

The first thing to get right is the question a site is trying to answer. It is not “human or bot.” Plenty of bots are welcome. Search crawlers, uptime monitors, and link previewers are all automation, and sites wave them through on purpose. The real question is narrower: does this traffic behave the way it claims to, and do its own signals agree with each other?

Detection is a hunt for contradictions. Every layer below is one more place where an automated client can accidentally tell two different stories at once, and every mismatch is a chance to get flagged. That framing matters, because it explains why the durable answer is honesty rather than disguise. If you are not pretending to be something you are not, there is no contradiction to catch.

Network and ip reputation

The earliest signal arrives before you send a byte of a real request: where you came from. Every connection has an ip address, and every ip sits inside a block that belongs to a network, identified by a number called an asn. An entire industry labels those blocks. This range is a mobile carrier, this one is home broadband, this one is a hosting company’s datacenter. Sites buy those labels, cache them right at the edge, and the check fires on the very first packet.

A request from a residential or mobile network starts with ordinary trust. A request from a cheap datacenter range starts under suspicion, because historically that is where automation lives. And the reputation is not really about you, it is about your neighbors. A fresh datacenter ip with no history of its own still inherits the cold reception of every scraper that ever ran from that provider. That is why pointing the cheapest pool of server ips at a strict site tends to fail fast. Nobody caught you doing anything wrong. The network you rode in on was already wearing the wrong label.

The tls handshake

Before any page loads, your client and the server negotiate an encrypted channel, and that negotiation is surprisingly loud. The exact way the software sets up tls, the cipher options it offers, the order it lists them in, and the extensions it includes all form a pattern. Real browsers produce well known patterns here, and they shift in predictable ways as browsers update.

A lot of scraping tools produce a tls pattern no real browser would send, because the underlying http library was never trying to imitate one. So a client can claim in its headers to be a current version of Chrome while its handshake quietly says it is a generic http library. That is a contradiction, and the edge exists to catch exactly that kind of mismatch.

Request headers and the user agent

Once the channel is open, the request itself carries more tells. A real browser sends a specific set of headers, in a specific order, with a user agent string that agrees with everything else. The field a site reads as your user agent is just text, so anyone can type Chrome into it. The rest of the request has to back that claim up.

Real Chrome sends particular hints about the platform, the language, and the way it handles compression and connections. An http client that sets a browser user agent and then forgets the dozen other things a browser would send is telling two stories again. Detection here is mostly consistency checking: do all the parts of this request describe the same thing they claim to be?

Browser and device fingerprinting

When a site wants a harder look, it runs code inside the browser itself, and this is where headless automation gets tested. A normal browser driving a real screen exposes thousands of small properties: the window size, the installed fonts, the way the gpu draws a shape onto a hidden canvas, the exact features webgl reports, the timing of tiny operations. Real devices vary across all of it in messy, natural ways.

An automation setup that spins up the same headless browser again and again tends to produce the same fingerprint every time, or a fingerprint carrying tells a normal machine would never have. The site is not hunting for one magic value. It is looking at the whole shape and asking whether it resembles a real person’s device or a cloned machine in a datacenter. There is also a quieter middle layer here: when a site is unsure, it hands the client a small challenge, a bit of work a real browser does without thinking. For a person it passes invisibly. For a crude client it is a wall.

Behavioral and rate signals

Past the technical fingerprints, sites watch behavior, and behavior is where scale gives automation away. A person browses in bursts, pauses, gets distracted, reads for a while, and comes back. A naive scraper pulls pages at a steady machine rhythm, one after another, faster than a human could read, at even intervals, around the clock with no sleep. No single request is strange. The shape over time is.

It widens out from one connection to the whole crowd, too. Edge networks sit in front of a huge slice of the web, so they see traffic for countless sites at once. That lets them spot a swarm of fresh visitors all taking the same path at the same speed, or many separate ips that each look fine alone but move in lockstep. Rate is part of it, how much you pull and how fast. Pattern is the deeper signal, and a new automation pattern caught on one site becomes a signature that defends the rest within minutes.

How compliant scrapers stay clean

So what does an operation that respects the rules do differently? It starts by reading the robots file and honoring it, treating the parts a site asks you to leave alone as off limits rather than as a suggestion. It identifies itself honestly where a site expects that, instead of dressing a server up as a browser it is not. It collects public data only, and stays away from anything behind a login, a paywall, or personal information that was never meant to be harvested.

The biggest lever is pace, and it is the one people skip because it feels slow. A compliant scraper runs at a rate the site can absorb, spaces its requests, backs off the moment it sees errors or slowdowns, and caches so it never asks twice for something it already has. It prefers an official api when one exists, because that is the front door the site actually built. Done this way, most of the detection stack never has a reason to fire, because you are not producing the contradictions it hunts for.

There is also a step above all the tooling: ask. If a site offers a data feed, a bulk download, or a documented api, that sanctioned path is almost always cheaper than fighting the detection stack. If the terms of service say no, then no clever handshake changes what you agreed to. For anything ambiguous, a short message to the site owner is worth more than a week of tuning a client, because permission turns the whole adversarial framing off.

The honest limits

I want to be straight about the boundaries. None of this makes anything undetectable, and I would not trust anyone who tells you it does. Detection keeps improving, the labels get sharper, and what passes quietly today can be flagged tomorrow. A clean network and honest behavior are not a trick that beats the wall. They are the absence of the contradictions that get you caught.

The moment your collection touches private data, or ignores what a site clearly asked you not to do, you have left the compliant lane entirely, whatever your tooling looks like. The durable version of this work is the boring version: public data, honest identification, a gentle pace, and the rules respected.

I run this infrastructure in production, so the frameworks, proxy setups, scraping apis, and honest tool reviews I write about are the ones I actually deal with. If you want the full written guides and the tools I use and test, read them at dataresearchtools.com. No undetectable promises, no guaranteed results, just the way this actually works.

Get new guides and videos first — join the Telegram channel.