OSINT Reconnaissance for Attack Surface Discovery

Methodology and tooling for open-source intelligence (OSINT) gathering as the first phase of an authorized security assessment: passive and active recon, subdomain enumeration, internet-wide scanning indexes, dorking, and attack surface management.

Introduction / Overview

Open-Source Intelligence (OSINT) is the practice of collecting and correlating information from publicly available sources - DNS records, certificate logs, search engine indexes, code repositories, social media, and internet-wide scan datasets - to build a picture of an organization's exposed attack surface. It is conventionally the first phase of any security assessment, executed before active scanning (Vulnerability Scanning) or exploitation attempts (Penetration Testing Methodology), because it establishes what exists before anyone attempts to determine whether it is vulnerable.

Scope and authorization. This document assumes an authorized, scoped engagement: a signed penetration test or red team agreement, an internal attack-surface-management (ASM) program run against your own organization, or reconnaissance conducted strictly within a published bug bounty program's defined scope. OSINT gathering on a target organization or individual without a legitimate, authorized purpose carries real legal exposure that varies significantly by jurisdiction (computer misuse, stalking, and data protection statutes all intersect with OSINT activity in ways that are not uniform globally), and it carries ethical exposure independent of legality - aggregating a private individual's personal data, movements, or affiliations without consent is stalking-and-harassment-adjacent behavior regardless of intent. Consult qualified legal counsel before any engagement whose authorization is ambiguous.

This document draws a hard line between two categories of OSINT that are frequently conflated:

• Organizational/technical OSINT - domains, IP ranges, exposed services, code repositories, cloud storage, employee counts and technology stacks used for social-engineering risk assessment. This is in-scope for essentially every mature security program and is the primary focus of this document.
• OSINT targeting private individuals - deep profiling of a specific person's identity, location history, relationships, or personal accounts. This requires materially stricter justification, explicit legal review, and in most organizations a distinct authorization path (e.g., insider threat investigations, executive protection) separate from a standard external assessment.

When to use this guidance: Scoping a new penetration test or red team engagement, standing up an internal attack surface management capability, conducting a pre-assessment reconnaissance phase, or evaluating third-party/vendor exposure as part of supply chain risk review.

Core Concepts

• Passive reconnaissance: Collecting information without any direct interaction with the target's infrastructure - querying third-party data sources (WHOIS registries, certificate transparency logs, search engine caches, internet-wide scan repositories) that have already collected the data. The target's systems see no traffic from the assessor and cannot detect the activity.
• Active reconnaissance: Directly interacting with target infrastructure - resolving DNS records against the target's own name servers, connecting to open ports, banner-grabbing a service. This produces fresher, more precise data but is observable by the target (firewalls, IDS, WAF logs) and crosses into territory that requires explicit authorization; see Vulnerability Scanning for the active-scanning phase that typically follows passive OSINT.
• Attack surface: The complete set of externally reachable assets (domains, subdomains, IP ranges, cloud services, exposed APIs, code repositories, employee-facing services) an adversary could target. OSINT's core deliverable is an accurate, current inventory of this surface - most organizations underestimate their own attack surface due to shadow IT, forgotten subdomains, and unmanaged cloud resources.
• Certificate Transparency (CT): A public-log requirement (mandated by all major browsers for publicly trusted TLS certificates) that every issued certificate be recorded in append-only, cryptographically verifiable logs. Because subdomains frequently appear in a certificate's Subject Alternative Name field before they are ever linked to publicly, CT logs are one of the highest-signal passive subdomain discovery sources available.
• OSINT aggregation vs. single-source lookups: A single WHOIS query or a single Shodan search answers a narrow question; realistic reconnaissance chains dozens of sources (DNS, CT logs, search engines, code repositories, breach databases) and correlates the results - which is why aggregation frameworks (theHarvester, SpiderFoot, Maltego) exist as a distinct tool category from single-purpose lookups.
• Attack Surface Management (ASM) as a discipline: the operationalization of OSINT into a continuous, repeated process rather than a one-time exercise performed at the start of an engagement - covered in its own section below.

Best Practices

1. Scope Explicitly Before Collecting Anything

Rationale: OSINT tooling makes it trivial to pull far more data than an engagement authorizes (employee personal social media, unrelated corporate subsidiaries, historical WHOIS registrant names). Defining scope in writing before the first query prevents both legal exposure and wasted effort on out-of-scope data.

In-scope:   *.example.com, AS64500 IP ranges, github.com/example-org repositories
Out-of-scope: employees' personal social media accounts, subsidiaries acquired after 2024
Authorization: signed SOW #2026-114, valid 2026-07-01 through 2026-08-15

Why this works well in production: A written scope document is the artifact that separates authorized reconnaissance from unauthorized surveillance, and it is the first thing a legal or compliance reviewer will ask to see if activity is ever questioned.

2. Start Passive, Escalate to Active Only With Explicit Authorization

Rationale: Passive sources (CT logs, WHOIS, cached search results) carry no detection risk and no legal ambiguity beyond "reading public data." Active techniques (DNS brute-forcing against the target's own resolvers, port scanning) generate logged traffic against the target's infrastructure and require the same authorization as the scanning phase covered in Vulnerability Scanning.

# Passive: query a third party that already scanned the internet
shodan search "hostname:example.com"

# Active: directly interact with target infrastructure (requires scope sign-off)
nmap -sV -p 443 203.0.113.10

Why this works well in production: Sequencing passive-before-active lets an assessor build a complete target inventory and prioritize what to actively touch, minimizing unnecessary direct interaction with production systems.

3. Correlate Across Multiple Independent Sources

Rationale: Every OSINT source has coverage gaps and staleness. Certificate transparency logs miss internal-only certificates from private CAs; DNS-based tools miss subdomains with no public DNS record but a live certificate; internet-wide scanners miss hosts behind CDNs that mask the origin IP.

subfinder -d example.com -silent > subs_subfinder.txt
amass enum -passive -d example.com -o subs_amass.txt
curl -s "https://crt.sh/?q=%.example.com&output=json" | jq -r '.[].name_value' | sort -u > subs_crtsh.txt
cat subs_subfinder.txt subs_amass.txt subs_crtsh.txt | sort -u > subs_combined.txt

Why this works well in production: Overlap between sources builds confidence in high-signal findings; discrepancies (a subdomain in crt.sh but not in DNS-based tools) often indicate decommissioned-but-still-certificated hosts, which are a common source of forgotten, vulnerable infrastructure.

4. Treat Findings as Time-Stamped, Not Permanent

Rationale: DNS records change, employees leave, certificates rotate, and cloud storage buckets are created and deleted continuously. A reconnaissance report from six months ago describes a point-in-time state, not the current attack surface.

Why this works well in production: This is the direct motivation for continuous ASM (see the dedicated section below) rather than a single point-in-time OSINT exercise - organizations that only reassess attack surface annually routinely discover forgotten subdomains and expired-but-still-DNS-resolvable infrastructure between assessments.

5. Minimize and Redact Personally Identifiable Data in Reports

Rationale: Reconnaissance on organizational assets frequently surfaces employee names, personal emails, or phone numbers as a side effect (WHOIS registrant contacts, document metadata authorship, LinkedIn enumeration). Retaining more of this than the engagement requires creates unnecessary privacy and legal risk for the assessor and the organization.

Why this works well in production: Reporting only what is necessary to demonstrate organizational exposure (e.g., "42 employee email addresses conforming to the discovered first.last@example.com format were found in breach corpora" rather than an enumerated list of names and personal accounts) keeps the deliverable focused on organizational risk rather than individual profiling.

Common Pitfalls & Anti-Patterns

Pitfall: Treating OSINT as Equivalent to Vulnerability Confirmation

Problem: Finding an exposed service via Shodan or an outdated software banner via a metadata leak is not the same as confirming an exploitable vulnerability - it is an indicator that warrants active verification.

Recommended approach: Feed OSINT findings into the active scanning and validation phase (Vulnerability Scanning) rather than reporting them as confirmed findings; see the "When Not to Use" section for the full comparison.

Pitfall: Unbounded Scope Creep During Aggregation Tooling Runs

Problem: Frameworks like SpiderFoot or Maltego will happily follow every discovered entity outward (a discovered employee's personal Twitter leads to their personal blog, leads to their home city) unless explicitly constrained, quickly drifting from organizational to individual-targeting OSINT.

Recommended approach: Configure module/transform allowlists restricted to organizational entity types (domains, IP ranges, corporate email formats) and explicitly exclude modules that pivot into personal social graphs unless that specific pivot was authorized in scope.

Pitfall: Relying on a Single Subdomain Enumeration Tool

Problem: No single passive subdomain tool has complete source coverage; relying on one (commonly Sublist3r alone, which has seen reduced maintenance activity relative to actively developed alternatives) systematically under-reports the true attack surface.

Recommended approach: Combine at least one certificate-transparency-based source, one multi-API aggregator (Amass or Subfinder), and a live DNS resolution pass to confirm which discovered names currently resolve, as shown in Best Practice #3.

Pitfall: Ignoring Rate Limits and Free-Tier Boundaries, Then Losing Access Mid-Engagement

Problem: Shodan, Censys, and similar SaaS OSINT sources meter access via credits or query counts; exhausting a free or low tier mid-engagement (common with Censys's 100 free credits/month or Shodan's 10,000 monthly API credits) stalls reconnaissance at an inconvenient point.

Recommended approach: Budget API credit consumption against the planned query volume before starting, and cache/deduplicate results locally so repeated pivots on the same target do not re-consume credits unnecessarily.

Pitfall: Assuming Internal/Corporate Assets Are Out of an Attacker's OSINT Reach

Problem: Teams sometimes assume that assets not linked from the corporate website are effectively invisible; certificate transparency logs, cloud provider IP range disclosures, and code repository history routinely reveal infrastructure never intentionally published.

Recommended approach: Assume any asset with a public IP address or a public TLS certificate is discoverable, and design the ASM program (see below) around continuously discovering what actually exists rather than what is intentionally documented.

Testing Strategies

OSINT reconnaissance is validated differently from application code - the relevant "tests" are process controls and data-quality checks rather than unit tests.

• Scope-compliance verification: Before and after each collection run, diff the assets touched/queried against the written scope document; flag any out-of-scope domain, IP, or individual that appeared in results for explicit review before inclusion in any report.
• Cross-source reconciliation: Automate a diff between subdomain/asset lists from independent sources (Amass vs. Subfinder vs. crt.sh) on every run; a source that suddenly returns zero new results relative to its historical baseline may indicate an API key expiration or rate-limit exhaustion rather than a genuinely smaller attack surface.
• Freshness/staleness checks: Tag every finding with its collection timestamp and source; re-verify DNS resolution and certificate validity for findings older than a defined threshold (e.g., 30 days) before including them in a final report.
• False-positive triage for internet-wide scan data: Shodan/Censys/ZoomEye results reflect the last time that provider's crawler observed a host and may include decommissioned services still holding a stale IP-to-hostname association; confirm liveness with a lightweight, explicitly authorized active check before reporting exposure.
• Chain-of-custody logging: Log every query issued (tool, source, timestamp, query string) during an engagement; this both supports scope-compliance review and lets an assessor reproduce a finding if a client disputes it.

Performance, Security & Scalability Considerations

• API credit economics dominate at scale: Internet-wide scan platforms meter by credit or query volume (Shodan's free tier: 10,000 monthly API credits, with paid tiers at $69/$359/$1,099 per month for higher volume; Censys's free tier: 100 credits/month, with purchased-credit "Starter" tiers valid 12 months). Running reconnaissance across hundreds of subsidiary domains requires either budgeting for a paid tier or building a caching layer that deduplicates queries across engagements.
• Local result caching reduces both cost and detection footprint: Persisting raw tool output (JSON/CSV) locally and re-querying only the delta since the last run scales better than re-running full enumeration on every assessment cycle, and reduces the number of queries against rate-limited free tiers.
• Data sensitivity of aggregated OSINT output: A consolidated reconnaissance dataset (subdomains, employee emails, breach exposure, technology stack) is itself a sensitive artifact - it is effectively a pre-built attack plan if it leaks. Store OSINT engagement output with the same access controls as a penetration test report, not as ordinary working notes.
• Horizontal scaling of continuous ASM pipelines: An internal ASM pipeline built from open-source tools (Amass, Subfinder, httpx for liveness, custom scheduling) scales by parallelizing per-domain or per-business-unit collection jobs and storing results in a queryable datastore (rather than flat files) once the organization's domain portfolio exceeds a few dozen properties.
• Commercial ASM platforms trade cost for coverage and automation: SaaS platforms in the CyCognito/Cortex Xpanse/Randori category perform continuous, algorithmic asset discovery (including "seedless" discovery that does not require the customer to enumerate their own domains first) and typically outperform a self-built open-source pipeline on discovery breadth for large, decentralized organizations, at materially higher cost than tool licensing alone.

Edge Cases & Advanced Usage

• CDN and WAF-fronted origins: Internet-wide scanners and passive DNS often resolve only the CDN edge IP, not the true origin server; origin discovery requires additional techniques (historical DNS records predating CDN adoption, SSL certificate reuse across the origin and a forgotten direct-access subdomain, or SPF/mail-record hints) and should be treated as a distinct, higher-effort sub-task.
• Cloud storage and serverless assets with no DNS footprint: S3-style object storage buckets, serverless function endpoints, and API gateway routes frequently have no corresponding subdomain and are missed entirely by DNS-based enumeration; dedicated cloud-asset discovery (provider-specific enumeration, bucket name permutation against known naming conventions) is a necessary complement, not a substitute, for subdomain enumeration.
• Certificate transparency false attribution: A CT log entry only proves a certificate was issued for a name - not that the name is currently in active use, or even that it was ever deployed; treat CT results as a candidate list requiring live resolution/liveness confirmation, not a confirmed asset list.
• Multi-tenant and shared-hosting ambiguity: An IP address discovered via Shodan/Censys may host dozens of unrelated tenants on shared infrastructure (shared hosting, PaaS multi-tenancy); attributing a discovered service to the target organization requires confirming ownership (matching TLS certificate SANs, HTTP response content, or WHOIS) rather than assuming IP-level exclusivity.
• Breach-data lookups for organizational exposure vs. individual targeting: Querying Have I Been Pwned's domain-search subscription tier for @example.com addresses is a legitimate organizational exposure assessment (identifying which employees' credentials appear in third-party breach corpora, informing a password-reset or MFA-enforcement decision); querying individual personal email addresses outside the organization's own domain crosses into individual-targeting OSINT and requires the stricter authorization path described in the Introduction.
• AI-assisted dork generation: Recent tooling increasingly generates Google-dork query variants from natural-language prompts; treat generated dorks the same as manually written ones - validate them against the actual GHDB category taxonomy before running them against a live target, since generated queries can be both overbroad (excessive noise) and underscoped (missing an established, higher-yield operator combination).

Domain, DNS, and Subdomain Enumeration

Subdomain enumeration finds the individual hostnames under an organization's registered domains that may host distinct, independently vulnerable services.

subfinder -d example.com -all -o subfinder_results.txt
amass enum -passive -d example.com -src -o amass_results.txt
curl -s "https://crt.sh/?q=%.example.com&output=json" | jq -r '.[].name_value' | sort -u

Internet-Wide Host, Device, and Service Discovery

These platforms continuously scan the public internet and index the results, letting an assessor query for exposure without touching the target directly.

shodan search "org:\"Example Corp\" port:443" --fields ip_str,port,org

Choosing among them: use Shodan or ZoomEye for the fastest turnaround on a known-exposure question; use Censys when result accuracy/freshness on a specific host matters most; use FOFA when the target has a significant presence in infrastructure that FOFA's query language surfaces well (component/certificate-based pivoting); combine at least two for a serious external exposure assessment, since no single index achieves full internet coverage.

WHOIS and Domain Registration Intelligence

whois example.com

Reverse WHOIS and historical WHOIS are the more valuable techniques for attack-surface work specifically, since they surface an organization's full domain portfolio (including domains registered by a marketing team or acquired subsidiary that IT has no record of) rather than confirming ownership of a domain already known.

Search-Engine-Based Reconnaissance (Dorking)

"Dorking" uses a search engine's own advanced operators to surface content the engine has indexed but that was never intended to be easily discoverable - exposed documents, admin panels, misconfigured directory listings, and leaked configuration files.

site:example.com filetype:pdf "confidential"
site:example.com inurl:admin
intitle:"index of" "backup" site:example.com

Roughly 58% of surveyed ethical hackers report using dorking as their first reconnaissance step, reflecting both its low cost and continued effectiveness for discovering exposed documents and misconfigured directory listings that never appear in a subdomain list.

Automated OSINT Aggregation Frameworks

theHarvester -d example.com -b crtsh,dnsdumpster,shodan -f example_harvest

Choosing among them: theHarvester and recon-ng suit fast, scriptable, CLI-native workflows; SpiderFoot suits broad automated coverage from a single seed with minimal manual configuration; Maltego suits engagements where the deliverable itself benefits from a visual entity graph (e.g., explaining organizational infrastructure relationships to a non-technical stakeholder).

Employee and Organizational OSINT for Social-Engineering Risk Assessment

This category assesses organizational exposure to social engineering and credential-based attacks - it is explicitly about the organization's risk posture, not about profiling specific employees as individuals.

• Company-page and LinkedIn enumeration: Identifying organizational size, technology stack (from job postings), and org-chart structure to assess phishing pretext plausibility and privileged-role targeting risk at the organizational level.
• Email format discovery/verification: Determining an organization's email naming convention (first.last@, flast@) via tools in the Hunter.io category enables an assessor to estimate the size of the phishable attack surface and construct a target list for an authorized phishing simulation - never for use outside the authorized engagement.
• Breach-data lookups for credential-exposure assessment: Have I Been Pwned's domain-search subscription (Core/Pro/High-RPM tiers; the underlying Pwned Passwords k-anonymity API remains free and unauthenticated) lets a domain owner enumerate which of their organization's email addresses appear in third-party breach corpora, directly informing password-reset and MFA-enforcement prioritization. This lookup is scoped to addresses under the organization's own verified domain - not a general-purpose lookup against arbitrary individuals.

curl -s "https://haveibeenpwned.com/api/v3/breacheddomain/example.com" -H "hibp-api-key: $HIBP_KEY"

Metadata Extraction from Public Documents and Images

Publicly posted documents and images routinely embed metadata revealing internal usernames, software versions, and file-system paths - the technique popularized by the FOCA tool against Microsoft Office and PDF documents.

exiftool -a -u -g1 downloaded_report.pdf

Findings from this category (internal usernames embedded in "last modified by" fields, printer paths, internal software version strings) feed directly into both the social-engineering risk assessment above (username format confirmation) and technology-stack fingerprinting for the active-scanning phase.

Code and Repository OSINT

Public code hosting is a high-yield, frequently underweighted reconnaissance source: leaked credentials, internal hostnames, and infrastructure references routinely persist in commit history even after a supposed "removal."

• GitHub/GitLab code search for leaked secrets or internal references: Search an organization's public repositories and, more importantly, forks and former-employee personal repositories for API keys, internal hostnames, and configuration file leakage. See Secure Coding Practices - secrets management for the corresponding prevention-side control, and Git Best Practices for why removing a secret from the latest commit does not remove it from history.
• Exposed .git directories and configuration files: Misconfigured web servers occasionally serve an application's .git directory or .env/config files directly; both GitHub dorking (filename:.env) and Shodan/ZoomEye web-content search can surface these at scale across many hosts simultaneously.

# GitHub code search (via the web UI or API) for a plausible internal reference
"internal.example.com" AND "password" org:example-org

Cross-reference every finding in this category against Secure Coding Practices - a leaked secret discovered via OSINT is direct, actionable evidence that the secrets-management controls described there have failed somewhere in the organization's pipeline.

Attack Surface Management as an Operational Discipline

OSINT reconnaissance, performed once, describes an attack surface at a single point in time. Attack Surface Management (ASM) is the discipline of running this same discovery process continuously, so that new subdomains, forgotten cloud assets, and expired-certificate hosts are surfaced as they appear rather than at the next scheduled engagement.

Building an internal ASM pipeline from the open-source tools covered above (Amass/Subfinder for subdomains, httpx for liveness confirmation, a scheduler, and a datastore for historical diffing) is a viable path for organizations with the engineering capacity to maintain it, and it keeps the full dataset in-house. Commercial ASM platforms trade that engineering investment for broader, often "seedless," algorithmic discovery and vendor-maintained data-source coverage, typically justified once an organization's domain portfolio and cloud footprint grow large enough that self-built pipeline maintenance becomes its own ongoing cost center.

When Not to Use / Alternatives

OSINT establishes what attack surface exists and provides context; it does not itself confirm that any discovered asset is exploitable.

Use OSINT alone when the objective is attack-surface inventory, third-party/vendor risk assessment, or continuous exposure monitoring where active interaction is out of scope or unauthorized. Escalate to active vulnerability scanning once assets are inventoried and authorization for direct interaction is confirmed. Escalate to full penetration testing when the objective requires demonstrating actual business impact (data exfiltration, privilege escalation, lateral movement) rather than confirming a vulnerability's theoretical presence - OSINT and scanning alone answer "what could be attacked and how," while penetration testing answers "what actually happens if it is."

Related Documentation

• Secure Coding Practices - secrets management and the prevention-side control for repository/code OSINT findings
• OWASP Top 10:2025 - the risk categories that discovered exposures typically map onto (misconfiguration, supply chain, access control)
• Vulnerability Scanning - the active-scanning phase that follows passive/organizational OSINT
• Penetration Testing Methodology - the full engagement lifecycle OSINT feeds into
• Git Best Practices - why secrets removed from a commit persist in history and remain discoverable

Glossary

• OSINT (Open-Source Intelligence): Information collection and analysis drawn exclusively from publicly available sources.
• ASM (Attack Surface Management): The continuous discipline of discovering, inventorying, and monitoring an organization's externally reachable assets.
• Passive reconnaissance: Information gathering with no direct interaction with target infrastructure; relies on third-party data sources.
• Active reconnaissance: Information gathering that directly interacts with target infrastructure (DNS queries against the target's own servers, port connections).
• Dorking: Using a search engine's advanced operators to surface indexed content not intended for easy discovery.
• GHDB (Google Hacking Database): A curated, categorized public repository of dork queries, hosted on Exploit-DB.
• Certificate Transparency (CT): A public, append-only log requirement for publicly trusted TLS certificates, widely used as a passive subdomain discovery source.
• EASM (External Attack Surface Management): The subset of ASM focused specifically on internet-facing (as opposed to internal-network) assets.
• k-anonymity API: An API design (used by HIBP's Pwned Passwords endpoint) that allows a client to check a value against a breach dataset by submitting only a partial hash prefix, so the full value is never transmitted or logged.
• Reverse WHOIS: A lookup that returns all domains registered under a given registrant attribute (name, email, organization) rather than looking up a single known domain.
• Seedless discovery: An ASM platform capability that discovers an organization's assets algorithmically without requiring the customer to supply an initial list of known domains/IP ranges.

Maintenance note: When updating this document, also update last_updated and verify current free/paid tier boundaries for Shodan, Censys, and Have I Been Pwned, as these change periodically.