Robots.txt

Hi Friends, Did you hear anything about this robots.txt file .This file is important for web development.The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocolis used to prevent files from web crawlers or Robots or Web Spiders .Web Crawlers ..What is this ? What is the purpose of this ? The answer is Web Crawlers areused(frequently used in search engines like Google ) to get data from World Wide Web .How Search engines work ??Alt textAlt Text 1. Web Spiders(Google) are getting the data from the World Wide Web and organizing the datain to their databases based on Meta Tags (this thing has been assigned for Index of this page).For example the web spiders get the meta tag as index(asp) and mapped to Asp.Net. 2. when we are searching the word ‘asp’ in Google , it searches the database(Google's Database-It was already filled by using Webcrawlers ) by using the keyword ‘asp’ and get theresults(in the order of maximum hit counts(Rank)) like Alt Text Now i am coming to that point robots.txt , this WebCrawler will get all datas from our web server.So we have to stop the web crawler to getting our personal datas or something you needto hide(example: login page) from our webserver, this is the time we need this Robot Exclusion Standard(robots.txt).Step 1: You can add the pages(need for hide ) one by one in the robots.txt file.Step 2: Upload this file to your Web Server.That's It…

This example allows all robots to visit all files because the wildcard "*" specifies all robots:

User-agent: *
Disallow:
This example keeps all robots out:
User-agent: *
Disallow: /

References: 1. Robots.txt 2. How Search Engine Works Examples: 1.Google robots.txt 2.Wikipedia robots.txt

comments powered by Disqus