SpiderFoot

SpiderFoot is an open source intelligence automation tool. Its goal is to automate the process of gathering intelligence about a given target.

Purpose

There are three main areas where SpiderFoot can be useful:

  1. If you are a security penetration tester, SpiderFoot will automate the reconnaisance stage of the test, giving you a rich set of data to help you pin-point areas of focus.

  2. Understand what your network/organisation is openly exposing to the outside world. Such information in the wrong hands could be a significant risk.

  3. SpiderFoot can also be used to gather threat intelligence about suspected malicious IPs, domain names or hosts you might be seeing in your logs or have obtained via threat intelligence data feeds.

Features

  • Utilises a shedload of data sources; over 50 so far and counting, including SHODAN, RIPE, Whois, PasteBin, Google, SANS and more.

  • Designed for maximum data extraction; every piece of data is passed on to modules that may be interested, so that they can extract valuable information. No piece of discovered data is saved from analysis.

  • Runs on Linux and Windows. And fully open-source so you can fork it on GitHub and do whatever you want with it.

  • Visualisations. Built-in JavaScript-based visualisations or export to GEXF/CSV for use in other tools, like Gephi for instance.

  • Web-based UI and CLI. Choose between a GUI that is easy to use and a powerful command-line interface. Take a look through the gallery for screenshots of the GUI and the collection of CLI videos on asiinema.org.

  • Highly configurable. Almost every module is configurable so you can define the level of intrusiveness and functionality.

  • Modular. Each major piece of functionality is a module, written in Python. Feel free to write your own and submit them to be incorporated!

  • SQLite back-end. All scan results are stored in a local SQLite database, so you can play with your data to your heart’s content.

  • Simultaneous scans. Each footprint scan runs as its own thread, so you can perform footprinting of many different targets simultaneously.

  • So much more.. check out the documentation for more information.

Data Sources

This is an ever-growing list of data sources SpiderFoot uses to gather intelligence about your target. A few require API keys but they are freely available.

Module Name Description
sfp_abusech.py abuse.ch Check if a host/domain, IP or netblock is malicious according to abuse.ch.
sfp_accounts.py Accounts Look for possible associated accounts on nearly 200 websites like Ebay, Slashdot, reddit, etc.
sfp_adblock.py AdBlock Check Check if linked pages would be blocked by AdBlock Plus.
sfp_ahmia.py Ahmia Search Tor ‘Ahmia’ search engine for mentions of the target domain.
sfp_alienvault.py AlienVault OTX Obtain information from AlienVault Open Threat Exchange (OTX)
sfp_alienvaultiprep.py AlienVault IP Reputation Check if an IP or netblock is malicious according to the AlienVault IP Reputation database.
sfp_archiveorg.py Archive.org Identifies historic versions of interesting files/pages from the Wayback Machine.
sfp_badipscom.py badips.com Check if a domain or IP is malicious according to badips.com.
sfp_base64.py Base64 Identify Base64-encoded strings in any content and URLs, often revealing interesting hidden information.
sfp_bingsearch.py Bing Some light Bing scraping to identify sub-domains and links.
sfp_bingsharedip.py Bing (Shared IPs) Search Bing for hosts sharing the same IP.
sfp_binstring.py Binary String Extractor Attempt to identify strings in binary content.
sfp_bitcash.py Bitcash.cz Malicious IPs Check if an IP is malicious according to Bitcash.cz Malicious IPs.
sfp_bitcoin.py Bitcoin Finder Identify bitcoin addresses in scraped webpages.
sfp_blockchain.py Blockchain Queries blockchain.info to find the balance of identified bitcoin wallet addresses.
sfp_blocklistde.py blocklist.de Check if a netblock or IP is malicious according to blocklist.de.
sfp_botscout.py BotScout Searches botscout.com’s database of spam-bot IPs and e-mail addresses.
sfp_builtwith.py BuiltWith Query BuiltWith.com’s Domain API for information about your target’s web technology stack, e-mail addresses and more.
sfp_censys.py Censys Obtain information from Censys.io
sfp_clearbit.py Clearbit Check for names, addresses, domains and more based on lookups of e-mail addresses on clearbit.com.
sfp_cookie.py Cookies Extract Cookies from HTTP headers.
sfp_crossref.py Cross-Reference Identify whether other domains are associated (‘Affiliates’) of the target.
sfp_crt.py Certificate Transparency Gather hostnames from historical certificates in crt.sh.
sfp_cybercrimetracker.py cybercrime-tracker.net Check if a host/domain or IP is malicious according to cybercrime-tracker.net.
sfp_cymon.py Cymon Obtain information from Cymon.io
sfp_dnsbrute.py DNS Brute-force Attempts to identify hostnames through brute-forcing common names.
sfp_dnsneighbor.py DNS Look-aside Attempt to reverse-resolve the IP addresses next to your target to see if they are related.
sfp_dnsraw.py DNS Raw Records Retrieves raw DNS records such as MX, TXT and others.
sfp_dnsresolve.py DNS Resolver Resolves Hosts and IP Addresses identified, also extracted from raw content.
sfp_dronebl.py DroneBL Query the DroneBL database for open relays, open proxies, vulnerable servers, etc.
sfp_duckduckgo.py DuckDuckGo Query DuckDuckGo’s API for descriptive information about your target.
sfp_email.py E-Mail Identify e-mail addresses in any obtained data.
sfp_errors.py Errors Identify common error messages in content like SQL errors, etc.
sfp_filemeta.py File Metadata Extracts meta data from documents and images.
sfp_fortinet.py Fortiguard.com Check if an IP is malicious according to Fortiguard.com.
sfp_fraudguard.py Fraudguard Obtain threat information from Fraudguard.io
sfp_freegeoip.py FreeGeoIP Identifies the physical location of IP addresses identified using freegeoip.net.
sfp_github.py Github Identify associated public code repositories on Github.
sfp_googlemaps.py Google Maps Identifies potential physical addresses and latitude/longitude coordinates.
sfp_googlesearch.py Google Search Some light Google scraping to identify sub-domains and links.
sfp_googlesearchdomain.py Google Search, by domain Some light Google scraping to identify sub-domains and links within site
sfp_hackertarget.py HackerTarget.com Search HackerTarget.com for hosts sharing the same IP.
sfp_honeypot.py Honeypot Checker Query the projecthoneypot.org database for entries.
sfp_hosting.py Hosting Providers Find out if any IP addresses identified fall within known 3rd party hosting ranges, e.g. Amazon, Azure, etc.
sfp_hostsfilenet.py hosts-file.net Malicious Hosts Check if a host/domain is malicious according to hosts-file.net Malicious Hosts.
sfp_hunter.py Hunter.io Check for e-mail addresses and names on hunter.io.
sfp_intfiles.py Interesting Files Identifies potential files of interest, e.g. office documents, zip files.
sfp_ipinfo.py IPInfo.io Identifies the physical location of IP addresses identified using ipinfo.io.
sfp_isc.py Internet Storm Center Check if an IP is malicious according to SANS ISC.
sfp_junkfiles.py Junk Files Looks for old/temporary and other similar files.
sfp_malc0de.py malc0de.com Check if a netblock or IP is malicious according to malc0de.com.
sfp_malwaredomainlist.py malwaredomainlist.com Check if a host/domain, IP or netblock is malicious according to malwaredomainlist.com.
sfp_malwaredomains.py malwaredomains.com Check if a host/domain is malicious according to malwaredomains.com.
sfp_malwarepatrol.py MalwarePatrol Searches malwarepatrol.net’s database of malicious URLs/IPs.
sfp_mcafee.py McAfee SiteAdvisor Check if a host/domain is malicious according to McAfee SiteAdvisor.
sfp_multiproxy.py multiproxy.org Open Proxies Check if an IP is an open proxy according to multiproxy.org’ open proxy list.
sfp_names.py Name Extractor Attempt to identify human names in fetched content.
sfp_nothink.py Nothink.org Check if a host/domain, netblock or IP is malicious according to Nothink.org.
sfp_onioncity.py Onion.city Search Tor ‘Onion City’ search engine for mentions of the target domain.
sfp_openbugbounty.py Open Bug Bounty Check external vulnerability scanning/reporting service openbugbounty.org to see if the target is listed.
sfp_pageinfo.py Page Info Obtain information about web pages (do they take passwords, do they contain forms, etc.)
sfp_pastebin.py PasteBin PasteBin scraping (via Google) to identify related content.
sfp_pastie.py Pastie.org Pastie.org scraping (via Google) to identify related content.
sfp_pgp.py PGP Key Look-up Look up e-mail addresses in PGP public key servers.
sfp_phishtank.py PhishTank Check if a host/domain is malicious according to PhishTank.
sfp_phone.py Phone Numbers Identify phone numbers in scraped webpages.
sfp_portscan_tcp.py Port Scanner - TCP Scans for commonly open TCP ports on Internet-facing systems.
sfp_psbdmp.py Psbdmp.com Check psbdmp.com (PasteBin Dump) for potentially hacked e-mails and domains.
sfp_pwned.py Pwned Password Check Have I Been Pwned? for hacked e-mail addresses identified.
sfp_ripe.py RIPE Internet Registry Queries the RIPE registry (includes ARIN data) to identify netblocks and other info.
sfp_robtex.py Robtex Search Robtex.com for hosts sharing the same IP.
sfp_s3bucket.py S3 Bucket Finder Search for potential S3 buckets associated with the target.
sfp_shodan.py SHODAN Obtain information from SHODAN about identified IP addresses.
sfp_similar.py Similar Domains Search various sources to identify similar looking domain names, for instance squatted domains.
sfp_social.py Social Networks Identify presence on social media networks such as LinkedIn, Twitter and others.
sfp_socialprofiles.py Social Media Profiles Identify the social media profiles for human names identified.
sfp_sorbs.py SORBS Query the SORBS database for open relays, open proxies, vulnerable servers, etc.
sfp_spamcop.py SpamCop Query various spamcop databases for open relays, open proxies, vulnerable servers, etc.
sfp_spamhaus.py Spamhaus Query the Spamhaus databases for open relays, open proxies, vulnerable servers, etc.
sfp_spider.py Spider Spidering of web-pages to extract content for searching.
sfp_sslcert.py SSL Gather information about SSL certificates used by the target’s HTTPS sites.
sfp_strangeheaders.py Strange Headers Obtain non-standard HTTP headers returned by web servers.
sfp_template.py Name Description
sfp_threatcrowd.py ThreatCrowd Obtain information from ThreatCrowd about identified IP addresses, domains and e-mail addresses.
sfp_threatexpert.py ThreatExpert.com Check if a host/domain or IP is malicious according to ThreatExpert.com.
sfp_tldsearch.py TLD Search Search all Internet TLDs for domains with the same name as the target (this can be very slow.)
sfp_torch.py TORCH Search Tor ‘TORCH’ search engine for mentions of the target domain.
sfp_torexits.py TOR Exit Nodes Check if an IP or netblock appears on the torproject.org exit node list.
sfp_torserver.py TOR Servers Check if an IP or netblock appears on the blutmagie.de TOR server list.
sfp_totalhash.py TotalHash.com Check if a host/domain or IP is malicious according to TotalHash.com.
sfp_uceprotect.py UCEPROTECT Query the UCEPROTECT databases for open relays, open proxies, vulnerable servers, etc.
sfp_virustotal.py VirusTotal Obtain information from VirusTotal about identified IP addresses.
sfp_voipbl.py VoIPBL OpenPBX IPs Check if an IP or netblock is an open PBX according to VoIPBL OpenPBX IPs.
sfp_vxvault.py VXVault.net Check if a domain or IP is malicious according to VXVault.net.
sfp_watchguard.py Watchguard Check if an IP is malicious according to Watchguard’s reputationauthority.org.
sfp_webframework.py Web Framework Identify the usage of popular web frameworks like jQuery, YUI and others.
sfp_websvr.py Web Server Obtain web server banners to identify versions of web servers being used.
sfp_whois.py Whois Perform a WHOIS look-up on domain names and owned netblocks.
sfp_wikileaks.py Wikileaks Search Wikileaks for mentions of domain names and e-mail addresses.
sfp_xforce.py XForce Exchange Obtain information from IBM X-Force Exchange
sfp_yahoosearch.py Yahoo Some light Yahoo scraping to identify sub-domains and links.
sfp_zoneh.py Zone-H Defacement Check Check if a hostname/domain appears on the zone-h.org ‘special defacements’ RSS feed.