OSINT Framework: Building a Complete Intelligence Pipeline

Blacksec · Jul 30, 2026

OSINT Framework - Building a Complete Intelligence Pipeline

1. Introduction
Open Source Intelligence (OSINT) is the backbone of modern reconnaissance. Every hack starts with information - domain names, email addresses, social media profiles, leaked credentials, metadata. The difference between a script kiddie and a professional operator is how thoroughly they map the target before sending a single packet.

This guide walks through building a complete OSINT pipeline - automated, modular, and repeatable. From passive reconnaissance to active scraping, from social media enumeration to dark web monitoring. By the end you will have a framework that runs on a cheap VPS and feeds intelligence into your attack workflow.

2. The OSINT Pyramid
Think of OSINT in layers:

Layer 1 - Passive Recon: No direct contact with target. DNS lookups, CT logs, WHOIS history, Google dorking.
Layer 2 - Semi-Passive: Indirect interaction. Shodan, Censys, Wayback Machine, social media API scraping.
Layer 3 - Active Recon: Direct but discreet. Port scanning, subdomain brute-force, directory enumeration.
Layer 4 - Aggressive: Full engagement. Vulnerability scanning, service fingerprinting, credential stuffing.

Most operators skip to Layer 3-4 and miss the goldmine in Layers 1-2. The best intel comes from public data the target forgot existed.

3. Tool Stack
Recon-NG: Swiss Army knife with 200+ modules covering DNS, geolocation, social media, credential harvesting, reporting.

recon/domains-hosts/certificate_transparency - Pull subdomains from CT logs
recon/companies-contacts/whois_pocs - Find contacts via WHOIS
recon/profiles-profiles/twitter_profile - Scrape Twitter
reporting/list - Export to CSV/HTML/JSON

TheHarvester: Email and subdomain enumeration from public sources.

Code:

theharvester -d target.com -b google,linkedin,bing,yahoo,baidu,crtsh

SpiderFoot: Automated OSINT scanner with 500+ modules. Runs as web service with cross-source correlation.

Code:

python3 sf.py -m all -s target.com -o html

Amass + Subfinder: Subdomain enumeration beasts. Feed them to each other for maximum coverage.

Code:

amass enum -d target.com -o subdomains.txt
subfinder -d target.com -all -o subs2.txt
cat subdomains.txt subs2.txt | sort -u > all_subs.txt

4. Building the Pipeline
Phase 1 - Domain Intel: CT logs (crt.sh), historical WHOIS (WhoisXML API), DNS brute-force with custom wordlists, zone transfer testing.
Phase 2 - People Intel: LinkedIn scraping, Github commit history for emails, PGP key servers, Google dorking (filetype

df "target.com" email), Sherlock cross-referencing.
Phase 3 - Infrastructure Intel: Shodan for IP ranges, Censys certificate analysis, cloud enumeration (AWS S3, Azure Blob), CDN detection, Wappalyzer fingerprinting.
Phase 4 - Leaked Data: Dehashed/IntelX API, dark web paste monitoring (AIL framework), Telegram channel scraping, RaidForums archive parsing.

5. Automation Script

Code:

#!/usr/bin/env python3
import os, json, subprocess, argparse
from datetime import datetime

class OSINTPipeline:
    def __init__(self, domain, out):
        self.domain = domain
        self.out = out
        self.results = {"domain": domain, "ts": str(datetime.now())}
    
    def phase1(self):
        subs = json.loads(subprocess.check_output(
            f"curl -s 'https://crt.sh/?q=%25.{self.domain}&output=json'", shell=True))
        self.results["subs"] = list(set(s["name_value"] for s in subs))
        subprocess.run(f"amass enum -d {self.domain} -o {self.out}/amass.txt", shell=True)
        subprocess.run(f"subfinder -d {self.domain} -o {self.out}/subs.txt", shell=True)
    
    def run(self):
        os.makedirs(self.out, exist_ok=True)
        self.phase1()
        with open(f"{self.out}/report.json", "w") as f:
            json.dump(self.results, f, indent=2)

if __name__ == "__main__":
    p = argparse.ArgumentParser()
    p.add_argument("-d", "--domain", required=True)
    p.add_argument("-o", "--output", default="./results")
    args = p.parse_args()
    OSINTPipeline(args.domain, args.output).run()

6. OpSec Considerations

Always route OSINT traffic through residential proxies for social scraping, datacenter for DNS lookups
Respect API rate limits - getting banned loses a data source permanently
Use throwaway VPS instances for aggressive scanning. Spin up, scan, destroy.
Store findings in structured DB (Elasticsearch or SQLite). Raw data is useless uncorrelated.
Never access OSINT resources from home IP. Never log into personal accounts from recon infra.

7. Google Dorking Cheatsheet

Code:

site:target.com intitle:index.of
site:target.com filetype:env DB_PASSWORD
site:target.com ext:xml phpinfo()
site:target.com inurl:wp-config.php
site:target.com ext:sql "INSERT INTO" "password"
site:target.com "BEGIN RSA PRIVATE KEY"
site:github.com "target.com" "password"

The best recon is the one the target never notices. Automate everything, document everything, build your attack from knowledge not guesses.

OSINT Framework: Building a Complete Intelligence Pipeline

Blacksec

Administrator