This release significantly re-structured the modules so that they are more data source-centric. This gives you much more control over what data sources are used during a scan in addition to making SpiderFoot even more configurable. Here are the details:
sfp_dns was split up into multiple modules to break up the tasks of simple DNS look-ups, DNS record querying, brute-forcing and look-aside queries.
sfp_malcheck was split up so that each module represented one data source. There was also a lot of clean-up to tailor the modules to the different types of data they can consume, and data they returned. Ultimately this will result in better data quality and improved performance.
sfp_blacklists was split up in the same way, but for blacklists like SpamCop, SORBS, etc.
sfp_sharedip is now split up to be per-data source, which is HackerTarget.com, Bing.com and Robtex.com.
sfp_pastes is also broken up to be specific to each data source. Notepadd.cc was dropped because it’s no longer alive.
Some modules were renamed to reflect their data source rather than something generic, so that it’s more meaningful when setting up a scan.
New modules and updates
New: sfp_builtwith uses the BuiltWith Domain API to identify names, e-mail addresses, web technologies, phone numbers and more.
New: sfp_ahmia: Searches the Ahmia.fi search engine to .onion sites (sites on the TOR network / dark web).
New: sfp_torch: Searches the TORCH search engine for .onion sites. You need TOR enabled for this to work, because TORCH is on TOR itself.
New: sfp_fraudguard: Queries the Fraudguard.io API to identify potentially malicious IPs, hosts and domains in addition to collecting Geo-IP information.
New: sfp_bitcoin: Identifies Bitcoin addresses on scraped web content.
New: sfp_blockchain: Queries blockchain.info to find the balance of identified Bitcoin wallet addresses that might have been scraped from web pages being spidered.
Update: Palevo tracker removed from sfp_abusech, as it’s no longer alive.
Update: sfp_robtex (previously part of sfp_sharedip) now uses the robtex.com API for idetifying co-hosted sites.
Enhancements / Bug fixes
CLI allows comments (prefixed by #)
Misc. minor bug fixes, performance improvements and tweaks.