Today I was trying to compare the anti robot rules between the major IoT search engine.
Starting from Shodan later Censys and ZoomEye and finalizing with Fofa.
As you can see in ZoomEye there are no rules.
Starting from Shodan later Censys and ZoomEye and finalizing with Fofa.
Shodan
User-Agent: Twitterbot Allow: /host/ # Every bot that might possibly read and respect this file. User-agent: * Crawl-delay: 10 Disallow: /search* Disallow: /host/ Disallow: /report/
Censys
User-agent: * Disallow: /ipv4 Allow: /ipv4/ Disallow: /ipv4/metadata Disallow: /ipv4/*/ Disallow: /certificates Allow: /certificates/ Disallow: /certificates/metadata Disallow: /certificates/*/ Disallow: /domain Allow: /domain/ Disallow: /domain/metadata Disallow: /domain/*/ Disallow: /data/scansio Disallow: /login Disallow: /logout Disallow: /account
ZoomEye
As you can see in ZoomEye there are no rules.
Fofa
# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file # # To ban all spiders from the entire site uncomment the next two lines: # User-agent: * # Disallow: /