Analyzing Robots.txt for Fun and Profit

Robots.txt is a plaintext file which is placed in the root directory ( abc.com/robots.txt ) of website and serves as an instruction set to automated web spiders / robots such as search engine bots (such as Googlebot and Yahoo Slurp!). Ideally all robots are supposed to obey the Robots Exclusion rules written in this file, but interestingly a lot of webmasters have misunderstood the intention of the file and are using it to hide confidential data and directories. In recent times this has made the robots.txt one of the first places for an Attacker to take a look at before he compromises a website's security. In this video we will explore how to interpret the robots.txt file (which is actually very simple) and look at a few examples of well known websites, which seeming have an interesting set of directories listed in their robots.txt file.

| View | Upload your own

Please download this file, before you view the presentation

Enable Javascript or Download Flash Playe if you see this!

Links:

1. Robots.txt

2. Robots and Spiders - 2600 Magazine

Vivek Ramachandran is a security evangelist and has been working in computer security related fields for the past 7 years. In 2007, Vivek spoke at world renowned conferences Defcon (WEP Cloaking Exposed) and Toorcon (The Caffe Latte Attack). The discovery of the Caffe Latte Attack was covered by CBS5 news, BBC online, Network World etc news agencies.In 2006, Vivek was announced as one of winners of the Microsoft Security Shootout contest held in India among 65,000 participants. He has also been a recipient of a Team Achievement at Cisco Systems for his work on 802.1x and Port Security modules on the Catalyst 6500 switches. Currently he spends all of his time maintaining Security- Freak.Net , SecurityTube.Net and is the founder of an online startup (currently in stealth mode). Vivek, is a Bachelor in Electronics and Communications Engineering from the prestigious Indian Institute of Technology, Guwahati.You can contact him at vivek[at]securitytube.net