A website checker is a mass status tool: feed it a list of URLs, it tells you which are alive, which are dead, which are behind challenges, and which are lying about their status through redirect chains and soft-404s. The naive version — fire a GET, read the status code, write live or dead — produces garbage at scale, because live/dead is not a status-code question: a 200 response on a parked-domain placeholder page is dead in every way that matters, a 403 from a WAF challenge is alive but protected, a 301 chain ending in a 200 through three hops needs all three hops resolved to judge the final destination, and a slow origin timing out at 10 seconds might be alive at 30. This is website-checker making decoded: the status semantics that actually classify live versus dead, the redirect and timeout discipline that keeps mass runs honest, the thread-pool and proxy architecture behind 10,000-URL batches, and the deployment stack (Flask or PHP front-ends with async back-ends) that open tooling consistently lands on.
ROUTE MAP (PLAIN ENGLISH)
The tool's job is classification, and the classification has more classes than live/dead. Alive (clean): a final 2xx after redirect resolution, response body above the parked-page threshold, server headers present. Alive-protected: 403/401/429 or a challenge-page signature (Cloudflare's cf-chl, Incapsula, PerimeterX markers) — the site is up, access is gated; conflating this with dead is the single most common mass-checker error because it sends perfectly good domains to the dead file. Redirected-alive: one or more 3xx hops resolving to a live final destination — the record carries the full chain so the checker's consumer sees where the URL actually lands (critical when checking redirector URLs or shortened links). Dead: connection refused, NXDOMAIN, definitive 404, or a 200 that fingerprints as parked (domain-squatter landing pages, "this domain is for sale" markers, empty-template signatures). Soft-dead: 5xx persisting past retry, origin timeouts past the retry budget, TLS handshake failures — likely dead or broken rather than parked, sorted separately because broken origins sometimes recover and parked domains never do. Unknown: network-layer errors from the checker's own side (local DNS failures, proxy-died mid-request) parked for re-run, never classified as dead on the tool's own failure.
STEP 1 - STATUS SEMANTICS THAT ACTUALLY WORK
STEP 2 - THE MASS ARCHITECTURE
STEP 3 - OUTPUT RECORDS AND CONSUMABILITY
STEP 4 - THE DEPLOYMENT STACKS THAT SHIP
WORKED EXAMPLE - A 5,000-URL BATCH
Input: 5,000 URLs, a mixed list of domain names and full URLs with heavy duplication across formats (http/https variants, trailing slashes, some shortened links). Normalization and dedupe collapse to 4,180 unique targets. Worker pool: 200 concurrent connections, 8-second connect timeout, 15-second read timeout, one retry on timeout/5xx. First pass results: 2,940 alive-clean (final 2xx with healthy body lengths), 412 alive-protected (mostly Cloudflare challenge signatures — their own file with cf-marker details), 187 redirected-alive (chains of 1-4 hops, final URLs recorded), 390 dead (connection refused or NXDOMAIN — the domain-list expired-registrations fraction), 145 parked-dead (200s fingerprinting as squatter pages — would have read alive in a status-code-only checker), 78 soft-dead (timeouts twice through), 28 unknown (proxy mid-request deaths — re-run after pool rotation drains to 3 on the second pass). Average latency: 840ms alive-clean, 9.2s soft-dead. The run report's one actionable line: protected-class concentrated on one CDN's edge range, meaning the batch's egress IPs drew the CDN's reputation eye — next run rotates egress before the protected-class inflates further. Total wall time: 14 minutes for the full batch with classification-complete records in every class file, final URLs where redirects moved, chains preserved.
WORKING SEQUENCE
QUICK ANSWERS
RELATED
The tool's job is classification, and the classification has more classes than live/dead. Alive (clean): a final 2xx after redirect resolution, response body above the parked-page threshold, server headers present. Alive-protected: 403/401/429 or a challenge-page signature (Cloudflare's cf-chl, Incapsula, PerimeterX markers) — the site is up, access is gated; conflating this with dead is the single most common mass-checker error because it sends perfectly good domains to the dead file. Redirected-alive: one or more 3xx hops resolving to a live final destination — the record carries the full chain so the checker's consumer sees where the URL actually lands (critical when checking redirector URLs or shortened links). Dead: connection refused, NXDOMAIN, definitive 404, or a 200 that fingerprints as parked (domain-squatter landing pages, "this domain is for sale" markers, empty-template signatures). Soft-dead: 5xx persisting past retry, origin timeouts past the retry budget, TLS handshake failures — likely dead or broken rather than parked, sorted separately because broken origins sometimes recover and parked domains never do. Unknown: network-layer errors from the checker's own side (local DNS failures, proxy-died mid-request) parked for re-run, never classified as dead on the tool's own failure.
STEP 1 - STATUS SEMANTICS THAT ACTUALLY WORK
- Method choice: HEAD first (half the bytes, the status and headers without the body), fall back to GET on targets that reject HEAD or return inconsistent HEAD/GET statuses — some origins 405 HEAD and 200 GET, some CDNs lie on HEAD; the two-step approach keeps mass throughput high while keeping classification honest for the minority of targets that misbehave on HEAD.
- Status-code mapping: 2xx → alive-candidate (awaiting body check), 3xx → follow (up to a hop cap, typically 5-10, with loop detection on repeated locations), 401/403 → alive-protected (record the auth-scheme header), 404/410 → dead, 429 → alive-rate-limited (site up, checker throttled — re-run class with backoff, not dead), 5xx → soft-dead-after-retry (one retry on a fresh connection before classifying), 000-class connection failures → refused (dead) or timeout (soft-dead-after-retry) by error type.
- Body-level truth: a 200 with a body under a threshold (a few hundred bytes) or matching parked-domain signatures ("domain is for sale", squatter-landing markers, template-empty heuristics) classifies parked-dead despite the 200 — the parked-page check is what stops domain-lists full of expired registrations from reading as fully alive; body checks only run on GET fallbacks, since HEAD carries no body.
- Challenge-page classification: Cloudflare, Incapsula, and PerimeterX challenge signatures (known marker strings and header patterns like cf-mitigated, server-challenge) classify alive-protected — the domain resolves, the origin answers, the edge is refusing unauthenticated clients; this class feeds its own output file because the consumer's next step (real browser, different network, timing) differs from dead-domain handling.
STEP 2 - THE MASS ARCHITECTURE
- Async worker model: an async HTTP client with a worker pool — not thread-per-URL (10,000 threads is a fantasy), but an async loop with a concurrency cap (hundreds of simultaneous in-flight requests per process, tuned against target politeness and the checker's own bandwidth); the queue drains the input list, each worker runs the two-step method flow (HEAD, GET fallback), classifies, writes to its class file with minimal lock contention (per-class buffered writers, not per-line file locks).
- Timeout budget: connect timeout and read timeout as separate knobs — connect around 5-10 seconds, read 10-30 depending on target class; the timeout choice directly decides soft-dead volume: aggressive timeouts flood the soft-dead file with slow-but-alive sites, lenient timeouts stall the queue behind dead origins; the two-timeout split lets connect-failures classify fast (dead/refused) while slow-reading origins get their full read budget.
- Retry policy: one retry on timeout and 5xx classes only, on a fresh connection; never retry 2xx/3xx/4xx classifications (they already have ground truth) and never retry through the same dead proxy (network-layer failures re-run on the worker's next proxy binding); the retry budget is what separates soft-dead (tried twice, still failing) from unknown (tool-side failure, re-run).
- Proxy and egress strategy: mass checking through shared egress gets rate-limited (429-class) or IP-banned at edge CDNs fast; the working patterns: residential proxy pool with per-worker binding for sensitive targets, direct egress with per-target pacing for politeness-sensitive batches, and domain-grouped scheduling (same-host URLs batched through the same worker so the checker's requests to one host look like a single visitor session, not a distributed hammer).
STEP 3 - OUTPUT RECORDS AND CONSUMABILITY
- Record shape: one line per URL carrying the full classification story — final URL after redirects, redirect chain count and hop list, final status code, classification class, response time, content-length where available, server header, challenge-type marker where protected, retry count; a consumer filtering live.txt needs the final URL (not the input URL when redirects moved), and a consumer debugging a soft-dead needs the response time and error type, which is why thin records ("url: dead") are useless at scale.
- Class files: alive.txt, protected.txt, redirected.txt (with chains), dead.txt, parked.txt, softdead.txt, ratelimited.txt, unknown.txt — the class taxonomy from step one mapped to one file each; unknown.txt gets re-run automatically after proxy-pool rotation, which usually drains it to near-zero on the second pass.
- Progress and rate surface: live counters (processed, per-class tallies, requests/second, current error streaks) — the operator surface that catches systematic failures in real time (a sudden 100% timeout streak means the egress died, not the internet); run reports at completion (totals per class, average latency per class, per-proxy contribution) feed the next run's tuning.
- Dedup and normalization: input URLs normalized before queueing (scheme defaulting, trailing-slash consistency, case-normalized hosts, fragment stripping) with duplicate collapse — the same URL appearing in five list variants otherwise produces five identical classifications and five x'd record count, which corrupts downstream statistics that assume one record per unique target.
STEP 4 - THE DEPLOYMENT STACKS THAT SHIP
- Flask pattern: Python async back-end (asyncio HTTP client) behind a Flask front-end — upload field for the URL list, config knobs (timeouts, concurrency, proxy toggle) as form controls, progress via server-sent events or polling, results downloadable as class-file bundles; the pattern dominates open tooling because the classification logic and the HTTP client live in the same language and the front-end is a thin shell.
- PHP/XAMPP pattern: PHP back-end with multi-curl (curl_multi handles the async layer in PHP-land) — the zero-install local pattern (XAMPP, drop folder in htdocs, browser open) that keeps the tool fully offline; multi-curl's callback model handles classification as each transfer completes; the pattern's ceiling is lower concurrency discipline than the Python async stacks, but for 1,000-5,000 URL batches it ships and runs anywhere PHP runs.
- CLI-first pattern: no front-end, batch runner with flags (input file, output dir, concurrency, proxy file, timeout) — the pattern that survives automation best (cron runs, pipeline integration, headless servers) — the web front-ends wrap the same core; authoring the core as a library first and the front-ends as shells keeps classification logic in one place instead of drifting between web and CLI versions.
- Deployment hygiene: rate defaults that respect targets (modest concurrency, honest user agents, the two-step HEAD-then-GET flow), output directories timestamped per run, config file for persisted knobs (proxy provider, timeouts, class thresholds) — the difference between a tool that gets used repeatedly and a toy that runs once.
WORKED EXAMPLE - A 5,000-URL BATCH
Input: 5,000 URLs, a mixed list of domain names and full URLs with heavy duplication across formats (http/https variants, trailing slashes, some shortened links). Normalization and dedupe collapse to 4,180 unique targets. Worker pool: 200 concurrent connections, 8-second connect timeout, 15-second read timeout, one retry on timeout/5xx. First pass results: 2,940 alive-clean (final 2xx with healthy body lengths), 412 alive-protected (mostly Cloudflare challenge signatures — their own file with cf-marker details), 187 redirected-alive (chains of 1-4 hops, final URLs recorded), 390 dead (connection refused or NXDOMAIN — the domain-list expired-registrations fraction), 145 parked-dead (200s fingerprinting as squatter pages — would have read alive in a status-code-only checker), 78 soft-dead (timeouts twice through), 28 unknown (proxy mid-request deaths — re-run after pool rotation drains to 3 on the second pass). Average latency: 840ms alive-clean, 9.2s soft-dead. The run report's one actionable line: protected-class concentrated on one CDN's edge range, meaning the batch's egress IPs drew the CDN's reputation eye — next run rotates egress before the protected-class inflates further. Total wall time: 14 minutes for the full batch with classification-complete records in every class file, final URLs where redirects moved, chains preserved.
WORKING SEQUENCE
Code:
website checker: normalize+dedupe input -> HEAD first, GET fallback on 405/inconsistent -> classify: 2xx alive-candidate (body check vs parked signatures), 3xx follow (hop cap + loop detect, chain recorded), 401/403/challenge-sig = alive-protected, 404/410 dead, 429 rate-limited re-run class, 5xx/timeout soft-dead after one fresh retry, conn-refused dead, local errors unknown (never dead) -> async worker pool, two-knob timeouts (connect vs read), per-worker proxy binding, domain-grouped scheduling -> class files + full records (final URL, chain, latency, markers) -> unknown re-run after egress rotationQUICK ANSWERS
- Why does a simple status-code checker produce garbage at scale? Because live/dead is not a status-code question — 200s on parked squatter pages are dead, 403s and challenge pages are alive-protected, redirect chains need resolution to judge the final destination, and tool-side network errors are unknown rather than dead; each of those classes needs its own handling downstream, which is why the classification taxonomy is the tool's real product and the status code is just one input to it.
- What is a soft-404 and why does it matter? A 200 response whose body fingerprints as a parked-domain placeholder ("domain is for sale", squatter-landing markers, empty-template signatures) — the status code says alive, the content says dead; at scale, expired domain lists are full of these, and a checker without body-level parked detection reports near-100% alive on lists that are mostly dead domains, which poisons every consumer decision downstream.
- HEAD or GET for mass checking? HEAD first — half the bytes, carries the status and headers needed for most classifications — with automatic GET fallback on targets that reject HEAD or return inconsistent HEAD/GET statuses (some origins and CDNs misbehave on HEAD); the two-step flow keeps throughput high while keeping classification honest for the minority of misbehaving targets.
- How should timeouts be configured? Connect and read as separate knobs: connect 5-10 seconds (decides refused-vs-dead fast), read 10-30 seconds (gives slow-reading alive origins their budget); aggressive single timeouts flood the soft-dead file with slow-but-alive sites, lenient ones stall the queue behind dead origins — the split plus one fresh-connection retry on timeout/5xx is what cleanly separates soft-dead from unknown.
- Why do challenge pages need their own class? A Cloudflare or Incapsula challenge means the domain resolves and the origin answers — the site is alive, the edge is refusing unauthenticated clients; conflating protected with dead sends good domains to dead-file consumers (who drop or replace them), and the correct downstream action (real browser, different egress, timing the retry) is completely different from dead-domain handling, which is why the challenge markers get extracted into the protected record.
- What concurrency is safe for mass checking? Async worker pools with a concurrency cap (hundreds in-flight per process, not thread-per-URL), per-target politeness via domain-grouped scheduling (one host's URLs batched through one worker so the checker looks like a single visitor, not a distributed hammer), and egress strategy matching target sensitivity (residential proxies for CDN-protected batches, direct with pacing for politeness-sensitive targets) — 429-class spikes mid-run mean egress reputation, not target death, and the re-run class catches it.
RELATED