A 404 page answers honestly

Sat, 03 Jul 2027 00:00:00 +0000

A 404 response has two audiences. Software needs an accurate status; a person needs a useful next step. A beautifully designed error page cannot compensate for a false 200, and a correct status does not excuse an inaccessible page. Keep both parts of the response truthful without turning a missing address into a business outage.

Choose the status before rendering

RFC 9110 says 404 means the origin found no current representation, or is unwilling to disclose that one exists. It does not establish whether the absence is permanent. Use 410 when removal is known to be likely permanent. For a documented move, redirect to the corresponding replacement; do not redirect every unknown URL to the homepage. That hides broken links and sends visitors somewhere unrelated.

A dependency outage is different from a missing resource. If a database failure prevents determining whether an object exists, preserve the appropriate server-error response instead of converting every exception into 404. Conversely, an authorization policy may deliberately conceal existence with 404. Document that decision without revealing identifiers, account membership, or permission details in the error body.

Google can classify a successful HTTP response containing an error message as a soft 404. A 200 does not guarantee indexing, and a 404 does not guarantee immediate disappearance from search results. The operational requirement is an honest status and useful content, not a promised crawler timetable.

Make the body independent and safe

Render a short explanation, a real home or index link, and enough context to recover. A static site need not add a search form or external widget just for this page. Keep its assets local and its operation independent of the database that may be failing. An API should return its documented machine-readable error format, not a successful HTML document.

Set the status before writing the body. In Go, the first body write implicitly selects 200 unless WriteHeader has already selected another status. HEAD should return the same status and applicable representation metadata as GET without transferring its content. Testing only a directly opened error-template file does not exercise the missing-resource route that invokes it.

When it breaks

A catch-all rewrite can serve the homepage for missing articles, JavaScript, or CSS files. A reverse proxy can replace the origin’s 404 with its own 200 error page. A custom handler can commit a successful header before discovering a missing file. Probe an invented path through the actual entry point, inspect the first response without following redirects, and separately check a known-good path.

Negative responses can also be cached: 404 is heuristically cacheable unless other rules or explicit controls prevent it. A long negative lifetime may hide a newly created or restored page. Choose the policy deliberately; no-store prevents new storage but is not a purge of an already stored response. Avoid echoing untrusted query strings into HTML or copying sensitive URLs into logs. Count normalized missing routes without creating unbounded metric labels.

Check it quickly

Save the following as not_found_probe.go in a scratch directory and run go run not_found_probe.go. It starts one loopback HTTP/1.1 server, follows no redirects, and modifies only an in-memory restoration flag. The deliberate /soft-miss bug is a negative control, not a recommended route.

package main

import (
 "fmt"
 "io"
 "net/http"
 "net/http/httptest"
 "runtime"
 "strconv"
 "strings"
 "sync/atomic"
 "time"
)

func must(err error) {
 if err != nil {
 panic(err)
 }
}

func check(ok bool, message string) {
 if !ok {
 panic(message)
 }
}

func main() {
 const missing = `<!doctype html><title>Not found</title><h1>Page not found</h1><a href="/">Home</a>`
 var restored atomic.Bool
 server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 status, body, media := 404, missing, "text/html; charset=utf-8"
 switch r.URL.Path {
 case "/":
 status, body = 200, "<!doctype html><title>Home</title><h1>Home</h1>"
 case "/gone":
 status = 410
 case "/moved":
 status = 308
 w.Header().Set("Location", "/")
 case "/offline":
 status, body = 503, "Service temporarily unavailable"
 case "/soft-miss":
 status = 200 // Deliberate bug: the body alone cannot identify success.
 case "/restored":
 if restored.Load() {
 status, body = 200, "Restored resource"
 }
 default:
 if strings.HasPrefix(r.URL.Path, "/api/") {
 media, body = "application/json", `{"error":"not_found"}`
 }
 }
 if r.Method != "GET" && r.Method != "HEAD" {
 status, body = 405, "Method not allowed"
 w.Header().Set("Allow", "GET, HEAD")
 if media == "application/json" {
 body = `{"error":"method_not_allowed"}`
 }
 }
 w.Header().Set("Content-Type", media)
 w.Header().Set("Cache-Control", "no-store")
 w.Header().Set("Content-Length", strconv.Itoa(len(body)))
 w.WriteHeader(status) // Status must be committed before writing the body.
 if r.Method != "HEAD" {
 _, err := io.WriteString(w, body)
 must(err)
 }
 }))
 defer server.Close()
 transport := &http.Transport{}
 defer transport.CloseIdleConnections()
 client := &http.Client{Transport: transport, Timeout: 2 * time.Second,
 CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse },
 }
 probe := func(method, route string, status int, media string) (*http.Response, string) {
 request, err := http.NewRequest(method, server.URL+route, nil)
 must(err)
 response, err := client.Do(request)
 must(err)
 body, readError := io.ReadAll(response.Body)
 closeError := response.Body.Close()
 must(readError)
 must(closeError)
 check(response.StatusCode == status && response.Proto == "HTTP/1.1", "wrong status or protocol")
 check(response.Header.Get("Content-Type") == media, "wrong media type")
 check(response.Header.Get("Cache-Control") == "no-store", "fixture must not reuse negative responses")
 check(!strings.Contains(string(body), "do-not-echo"), "request query leaked into error page")
 fmt.Printf("%s %s: %d, body=%d bytes\n", method, request.URL.Path, status, len(body))
 return response, string(body)
 }
 const html = "text/html; charset=utf-8"
 probe("GET", "/", 200, html)
 get, body := probe("GET", "/missing?hint=do-not-echo", 404, html)
 check(body == missing && strings.Contains(body, `href="/"`), "missing page must offer a real home link")
 head, empty := probe("HEAD", "/missing", 404, html)
 check(empty == "" && head.ContentLength == get.ContentLength, "HEAD must preserve status and length without a body")
 probe("GET", "/assets/missing.css", 404, html)
 _, api := probe("GET", "/api/missing", 404, "application/json")
 check(api == `{"error":"not_found"}`, "API must not receive an HTML success page")
 probe("GET", "/gone", 410, html)
 move, _ := probe("GET", "/moved", 308, html)
 check(move.Header.Get("Location") == "/", "known move must name its target")
 probe("GET", "/offline", 503, html)
 soft, softBody := probe("GET", "/soft-miss", 200, html)
 check(softBody == missing && soft.StatusCode != get.StatusCode, "negative control must expose a soft 404")
 probe("GET", "/restored", 404, html)
 restored.Store(true)
 _, recovered := probe("GET", "/restored", 200, html)
 check(recovered == "Restored resource", "restored resource must no longer return 404")
 rejected, _ := probe("POST", "/missing", 405, html)
 check(rejected.Header.Get("Allow") == "GET, HEAD", "method rejection must describe allowed methods")
 apiRejected, apiError := probe("POST", "/api/missing", 405, "application/json")
 check(apiRejected.Header.Get("Allow") == "GET, HEAD" && apiError == `{"error":"method_not_allowed"}`, "API method errors must retain JSON")
 fmt.Println("PASS: thirteen requests; missing, moved, unavailable, and restored remain distinct")
 fmt.Println(runtime.Version())
}

The thirteen requests distinguish a healthy homepage, HTML and API misses, a missing asset, known removal, a known move, and temporary unavailability. GET and HEAD agree on the missing resource’s status and declared length, but HEAD has no body. The query marker never appears in the response. The soft-miss body is identical to the genuine error page while its 200 status exposes the defect. Restoring a resource changes its answer from 404 to 200; POST is rejected separately with 405 and Allow, and an API method error retains its JSON format.

The assertions passed with Go 1.27.0 on macOS. The fixture checks response construction, not crawler indexing, CDN negative-cache invalidation, or a production proxy configuration. Before deployment, repeat the good-path and invented-path checks against the built site’s real serving stack in both languages.

Sources

RFC 9110: HTTP Semantics, 404, 410, HEAD, and representation semantics, checked 2026-09-09.
RFC 9111: HTTP Caching, negative response storage and cache controls, checked 2026-09-09.
MDN: 404 Not Found, missing resources and custom error pages, checked 2026-09-09.
Google Search: HTTP and network errors, soft 404s and indexing behavior, checked 2026-09-09.
Go net/http, response status commitment and redirect control, checked 2026-09-09.