The Path Not Found Is the Point

This site's robots.txt asks crawlers to stay away from six paths. None of the six exist. They have never existed, and nothing on the site links to them.

For a legitimate crawler, a Disallow rule is a standard exclusion directive, and honouring it costs nothing. For a scanner, the same file reads as a list of places worth looking. That asymmetry is the trick, and it is an old deception pattern: seed robots.txt with paths that only a scanner would follow, and every request for one becomes a self-identification. Either a bot read the file and ignored the directive, or the path came from a scanning wordlist. No reader arrives there by accident.

Those six paths are the exact-match half of the site's probe filter. Four wildcard patterns cover common scanning prefixes. In the Athena queries that read the CloudFront access logs, the whole filter is one condition:

AND (
  uri IN ('/.env', '/admin/', '/wp-admin/', '/wp-login.php', '/api/', '/config/')
  OR uri LIKE '/wp-%'
  OR uri LIKE '%xmlrpc%'
  OR uri LIKE '%/admin.php%'
  OR uri LIKE '%/sc.php%'
)

A request matching it is classified as a known probe, and the results surface through the site's security dashboard.

Why this works better on a static site#

Most honeypots earn their signal by pretending to be vulnerable: a fake SSH service, a decoy login page, an application that looks exploitable. An attacker interacts with the decoy, and the system records what they do.

This site has none of that; what it offers instead is an unusually clean baseline. With no dynamic application surface, a request for a path that was never part of the site has few honest explanations.

CloudFront access logs already capture each request: path, status code, timestamp, edge location, and related metadata. The analysis lives in the logging pipeline. The site itself does nothing, which suits it, since doing nothing is also its entire defence.

Two tiers of observation#

Ten patterns cannot be exhaustive, so requests that fall on paths outside the curated filter and return 4xx form a second tier of novel probes. These could be scanning campaigns the filter has not caught up with, or something else entirely, including the occasional mistyped URL. For the novel tier, the pipeline collects only aggregates: path, hit count, and unique IP count. It never selects client_ip or user_agent for an individual request.

Confidence and collection are tied together. A request matching the curated filter is recorded in full: IP address, user agent, timing, edge location. A request in the novel tier is counted but never identified. The condition shown above is shared across the probe queries, including the one that selects client_ip and user_agent, so the clause that decides what counts as a probe also decides whose individual data gets recorded. It is a detector and a collection boundary at once, and widening the filter to catch more scanning also widens the set of requests whose individual data is kept. That coupling is the thread the rest of this series follows.

The first post showed that a significant fraction of probe traffic falls outside the curated filter. That is the design working as intended. The novel tier surfaces what the filter misses, at a lower level of observability, and between them the two tiers produce a view of automated scanning without identifying the people who actually read the site.

What this shows#

The whole observation system is ten patterns and a second tier of counters, built on logs the site was already producing. No agent, no appliance, no subscription or third-party code. The machinery is close to nothing, and so is the bill.

A single snapshot only shows what the filter caught on one pass. Over time, the same setup can show which probes persist, which disappear, and where commodity scanning moves next.