Why S3 Returns 403 When It Means 404

While reviewing the site's analytics dashboard, I noticed a handful of 403 responses for paths that clearly did not exist. These were broken links to an article I hadn't published yet. The expected response was 404. Instead, S3 was actively refusing the request.

This wasn't an access control misconfiguration. It was S3 behaving exactly as designed.

The behaviour#

When CloudFront requests an object from S3 via Origin Access Control (OAC), S3 evaluates the request against the bucket policy. A typical static site policy grants s3:GetObject on the bucket's objects:

statement {
  actions   = ["s3:GetObject"]
  resources = ["${aws_s3_bucket.site.arn}/*"]
 
  principals {
    type        = "Service"
    identifiers = ["cloudfront.amazonaws.com"]
  }
 
  condition {
    test     = "StringEquals"
    variable = "AWS:SourceArn"
    values   = [aws_cloudfront_distribution.site.arn]
  }
}

This is sufficient for serving existing objects. But when the requested object doesn't exist, S3 needs to decide what to tell the caller. If the caller has s3:ListBucket permission on the bucket, S3 returns 404. If not, S3 returns 403.

The reasoning is straightforward: without permission to list the bucket's contents, the caller shouldn't be able to determine whether a given key exists. A 404 would confirm that the key does not exist, which is itself information about the bucket's contents. So S3 returns 403 instead, treating "does this object exist?" as an unauthorised question.

The fix#

The fix has three parts.

1. Grant ListBucket#

Adding a s3:ListBucket statement to the bucket policy allows S3 to return honest 404 responses. Note that ListBucket targets the bucket ARN, not the objects path:

statement {
  actions   = ["s3:ListBucket"]
  resources = [aws_s3_bucket.site.arn]
 
  principals {
    type        = "Service"
    identifiers = ["cloudfront.amazonaws.com"]
  }
 
  condition {
    test     = "StringEquals"
    variable = "AWS:SourceArn"
    values   = [aws_cloudfront_distribution.site.arn]
  }
}

This doesn't expose bucket contents to end users. CloudFront only supports GET and HEAD methods in this configuration. There is no mechanism for a viewer to issue a ListObjects request through CloudFront. The permission is used internally by S3 when CloudFront requests a non-existent key. ListBucket is evaluated at the bucket level and does not grant object retrieval.

2. Add a CloudFront custom error response#

Without a custom error response, CloudFront passes through whatever S3 returns for a 404. For a static site that means raw XML. Adding a custom_error_response block tells CloudFront to serve a specific page instead:

custom_error_response {
  error_code            = 404
  response_code         = 404
  response_page_path    = "/404.html"
  error_caching_min_ttl = 60
}

The error_caching_min_ttl controls how long CloudFront caches the error response at edge locations. Sixty seconds is a reasonable default: short enough that deploying a previously missing page takes effect quickly, long enough to absorb repeated requests for the same broken path.

3. Add a 404 page#

For a Next.js static export, creating app/not-found.tsx in the App Router generates a 404.html in the build output. This is the file CloudFront serves via the error response above.

The page itself is minimal: a heading, a short message, and links back to useful parts of the site (home, posts, search). Nothing elaborate. The goal is to acknowledge the dead end and offer a way out.

The result#

After deploying these three changes, requests for non-existent paths return a 404 with a styled page instead of a 403 with S3's XML error body. The analytics now accurately distinguish between genuine access denials and missing content, which restores the intended distinction between missing content and access denial.

Addendum: the effect in the data#

The analytics dashboard makes the change visible. The status codes chart shows 403 responses climbing steadily through 22 February, peaking around 950 requests, then collapsing on the 23rd as the fix went live. 404s briefly spiked on the same day as those requests were reclassified, then tapered off as the scanning traffic moved on.

(click to enlarge)

The crossover is clean enough that the deployment date is identifiable from the chart without needing to check the deployment logs.