Web Archives and Domain Intelligence

Web Archives and Domain Intelligence
Field	OSINT; digital evidence preservation
Archives	Wayback Machine (1 trillion+ pages); archive.today (contested)
Registration data	RDAP (RFC 9082/9083), replacing WHOIS for gTLDs since 28 Jan 2025
Infrastructure data	Passive DNS; Certificate Transparency (RFC 9162)
Evidentiary concern	Authentication, completeness, chain of custody

Web archives preserve past states of websites, and domain intelligence is the analysis of domain registration, DNS, certificate and hosting records to identify who operates a site and what else they run. In fraud work the pair answers: what did this site claim before it was edited or taken down, when did it appear, and what other sites share its infrastructure? Fraudulent investment platforms, fake charities and phishing kits are short-lived and frequently rewritten, so a site that cannot be revisited tomorrow must be captured today. This article treats methods and evidence handling; it supports work on entities and fraud infrastructure, not on private individuals (see OSINT Ethics).

1 Web archives

The Internet Archive’s Wayback Machine has operated since 1996 and in October 2025 announced that it had archived more than one trillion web pages, preserving hundreds of millions more each day.^[1] Captures are made by its crawlers, by partner and user-submitted jobs, and on demand by the “Save Page Now” feature. Coverage is uneven: popular pages are captured often, obscure ones rarely; some assets (scripts, images, embedded frames) may be missing or captured at different times from the page, so a rendered snapshot can mix content from different dates. Pages blocked by robots rules, login walls or JavaScript-heavy applications may be incomplete.

archive.today (archive.is and sibling domains) takes on-demand snapshots and is widely used for paywalled or rapidly changing pages. Its reliability is now contested: in 2026 English Wikipedia deprecated it, citing evidence that its operators had embedded code used in a distributed denial-of-service attack and had altered content of archived pages, and the domains were globally blacklisted by Wikimedia stewards in June 2026.^[2] For an evidentiary file, prefer multiple independent captures (Wayback plus your own forensic capture) and do not rely on archive.today alone.

2 Registration data: WHOIS and RDAP

Domain registration data historically came from the WHOIS protocol. On 28 January 2025 the contractual requirement for most generic top-level domain registries and all gTLD registrars to offer WHOIS service was removed, making the Registration Data Access Protocol (RDAP), defined in RFC 9082/9083, the definitive source: it returns structured JSON over HTTPS and supports differentiated access.^[3] Since the GDPR era, most registrant names, emails and phone numbers are redacted, so current records usually yield the registrar, creation and expiry dates, name servers, status codes and sometimes an organisation field. Country-code TLD registries follow their own policies. Historical WHOIS snapshots held by commercial providers can reveal pre-redaction registrant details, but their accuracy and licensing should be checked, and registrant data of individuals should be handled with care.

3 Passive DNS and certificate transparency

Passive DNS databases record DNS answers observed by sensors over time, letting an investigator see which IP addresses a domain has resolved to, which other domains have shared an IP, and when changes occurred, without touching the target. Certificate Transparency (CT), specified in RFC 9162, requires publicly trusted TLS certificates to be recorded in public append-only logs; browsers such as Chrome require Signed Certificate Timestamps from approved logs.^[4] Search interfaces such as crt.sh let anyone list every certificate issued for a domain, including subdomains that were never linked publicly, with issuance dates that often precede a site’s launch.

Hosting and analytics pivots. Websites leak shared identifiers: the hosting IP and ASN, name servers, favicon hash, page title, HTML structure, Google Analytics or tag-manager IDs, advertising IDs, contact emails, payment-processor and chat-widget keys. Searching for a shared ID across crawled datasets can reveal a family of sites run by one operator, which is how many scam networks are mapped. Shared hosting and CDN fronting produce false links, so a shared IP alone is weak.

4 Forensic examination techniques

Technique 1: Capture-history timeline. List every Wayback capture of a URL (the CDX index returns timestamps, status codes and content digests). Compare digests: identical digests mean unchanged content; changes mark edits. Worked example: 140 captures of a “trading platform” homepage from 12 March to 30 September show the same digest until 17 June, when the claimed licence number, address and “since 2012” founding claim all change. Output: a table of capture dates, digest changes and what changed, showing when claims were altered.

Technique 2: Claim-versus-infrastructure test. A site claiming a decade of operation should show an old domain. Example: the site says “established 2012”, but RDAP shows creation 21 days before the first CT certificate and first Wayback capture 9 days later. The domain age (creation date) is checked against the claim, and also against the first appearance in passive DNS and the earliest certificate: three independent clocks. Disagreement between claim and clocks is a documented finding. Note that a domain can be bought aged, so age alone does not prove history.

Technique 3: Certificate-based discovery. Query CT for the domain and for the organisation name, then extract subdomains and the dates of issuance. Look at issuance clustering: 30 certificates for different brand-themed domains issued within one week from the same issuer account pattern suggests a templated deployment. Output: a subdomain and certificate inventory with log IDs.

Technique 4: Infrastructure pivot with a confidence score. Build a table of candidate domains and for each record shared attributes: IP (weak), name server (weak to moderate), analytics ID (strong), unique contact email (strong), favicon hash (moderate), identical code or text (moderate). Score each pair by the number of strong independent links. Example: Site A and Site B share a name server and an IP (weak, hosting overlap), but also an identical analytics ID and an identical refund-policy text: two independent strong links, so assess them as probably operated together. Document negative checks too.

Technique 5: Content diffing. Save two captures and compare their text and HTML sources line by line to see exactly what was removed, such as risk disclaimers, named staff, or a regulator-registration claim; then verify the claim at the named regulator’s own register. Output: a diff exhibit with hashes of both captures.

Technique 6: Lifecycle correlation. Align domain registration, CT issuance, first capture, advertising activity, complaint dates and takedown. Fraud sites cluster in a short life cycle; repeated registration of successor domains with the same template is a recidivism indicator.

5 Preservation and evidentiary capture

Capture must be done before the page changes and must be reproducible. A defensible procedure:

1. Record the context: date and time with time zone, examiner, workstation, network, browser version, the exact URL and how it was found.
2. Capture several forms: a full-page screenshot, the saved HTML and assets (for example a WARC file made with a recognised capture tool), a PDF print, and the HTTP response headers.
3. Hash each file (SHA-256) at capture and record the hashes in a log; store the originals read-only and work on copies.
4. Submit an independent archive copy to the Wayback Machine, noting that a public capture is not private and may alert the operator.
5. Keep a log of every step so another examiner can repeat it; this is the heart of chain of custody and the digital forensic lifecycle.
6. Capture from a controlled, clean environment so the operator’s tracking of the visit does not compromise the investigation; prefer passive sources first.

On admissibility, US federal evidence rules allow certain electronic records to be authenticated by a qualified person’s certification (Fed. R. Evid. 902(13) and 902(14)), but courts differ about archive captures, and it is common to supplement with a witness who can testify to how the capture was made; see Evidence and Testimony. Consult counsel for the jurisdiction. A hashed forensic capture made by the investigator is usually stronger than a third-party archive link.

Source	Shows	Weight / caveat
Wayback capture	Past page state, date	Independent third party; mixed-date assets possible
archive.today	On-demand snapshot	Contested reliability after 2026 findings
RDAP record	Registrar, dates, name servers	Registrant usually redacted
CT log entry	Certificate issuance date and names	Strong timestamp; shows names, not owners
Passive DNS	Historical resolution	Sensor coverage limited; shared hosting false links
Own forensic capture	State at time of capture	Strongest if hashed and logged

6 Limits

• Attribution. Infrastructure links show common technical control, not identity; avoid naming individuals without corroboration from lawful process such as registrar or hosting-provider disclosure.
• Privacy. Redaction exists for a reason; do not attempt to unmask private registrants.
• Tool lock-in and opacity. Commercial passive-DNS data cannot always be independently verified; state the vendor and date.
• Manipulation. Operators can forge analytics IDs, use privacy proxies, hide behind CDNs and request archive exclusion.
• Active contact. Visiting a site is a form of contact (see passive versus active reconnaissance); logging in, submitting forms or probing for vulnerabilities is outside OSINT.

See also: OSINT Tools, Digital Forensics, Cloud and IoT Forensics, Geolocation and Imagery OSINT, Corporate Registries and Leak Databases.

References

1. Internet Archive. “One Trillion Web Pages Archived.” 31 October 2025. blog.archive.org
2. Wikimedia Meta-Wiki. “Archive.today incident.” meta.wikimedia.org
3. ICANN. “Launching RDAP; sunsetting WHOIS.” 27 January 2025. icann.org
4. Certificate Transparency (RFC 9162 overview). en.wikipedia.org