Thursday, April 16, 2015

What is Robots.txt?

Every blogger should have to add robots.txt file on their blogger blog to manage their content on search engine. If you don't manage your blog through robots all the peoples can get access to your private pages, which you don't have to show to all people.

Robots.txt is a text file which contains lines of simple code to manage blogger content on search engines. It is saved on the website or blog’s server which instruct the web crawlers to how to index and crawl your blog in the search results.

Lets say if you instruct your robot to restrict any web page on your blog from web crawlers that can't get indexed in search engine. 

Each Blogger blog will have a robots.txt file that comes by default and it looks something like the one below. You can check your blogs robots.txt file by adding /robots.txt next to your domain name. ( or if you have added custom domain on your blogger blog then you can go through this (


User-agent: Mediapartners-Google
User-agent: *
Disallow: /search
Allow: /
If you have added custom domain on your blogger blog then you can replace sitemap section with your current domain name. i.e,

If you have XML site map then again you can replace your sitemap section i.e with thats it.

Explanation of Robots.txt Terms,

User-agent: Mediapartners-Google – This terms for Google adsense that is used to server better relevant ads on your site based on your content.

User-agent*: User-agent is marked with the * sign, this will not index all the labels exist on blogger. It will index all the pages which are exist on blogger.

Disallow: /search : Disallow search will ignore those URLS which are having /search keyword just after the domain name.
Allow: / : Allow refers to those blog pages  which are specially allowable to index in search engine.

Sitemap: Sitemap will help to crawl all the pages on blog and help to index in search engine.  

How to Add Custom Robots,

  • Go to blogger Dashboard, Now go for Setting and Click on Search Preference, Now its time to scroll and you will get Crawlers and indexing section, There you will fine Custom robots.txt Go for edit and click on Yes button to add custom robots txt. 
Add Custom Robots
  • Now copy above robot.txt code and paste it on blank space. Then you just have to change at the sitemap section with your sitemap file. that's it. 

AnsMachine . 2017 Copyright. All rights reserved. Designed by Blogger Template | Free Blogger Templates