Importance Of Robots.txt and How To Improve Your SEO With Robots.txt

So today we will discuss about a very important factor that can plays a really important role in getting a good SEO for you blog or website. The robots.txt file is a very important file if you want to have a good ranking on search engines. I have seen many websites that don't offer the robots.txt file but I recommend this that every website or blog should have this file. Having robots.txt file is a positive feature for your website or blog and it helps in Serach Engine Optimization. If your Web site doesn't have a robots.txt file yet, read on to learn how to create and use robots.txt file. 

You might also like this :: How To Use Custom Robots Header Tags In Blogger

You might also like this :: How To Add Google Custom Search Box To Your Blog Or Website

What is Robots.txt ?

Robots.txt file tells to the search engine spider, which of the Web Pages of your site should be indexed and which of the Web pages should be ignored. By default the search engine bots crawl everything possible unless they are forbidden from doing so. They always scan a file that is called robots.txt file before crawling the web site.

The robots.txt file is a simple text file as I told you above and make sure that it should be placed in your root directory, for example:

                 http://www.yourwebsite.com/robots.txt

How To Create Robots.txt File ?

Before going to How To Create Robots.txt lets have a small introduction about the terms that are used in robots.txt file ::

now after knowing the meaning of each term that are use in robots.txt file lets create a robors.txt file :

You can check many other sites' robots.txt to get a rough idea on how these are generally implemented.

• http://www.cnn.com/robots.txt
• http://www.nytimes.com/robots.txt
• http://www.spiegel.com/robots.txt
• http://www.ebay.com/robots.txt

From the Editor’s Desk

As I told you above that robots.txt file plays a very important role for good SEO and if you want to have good rankings on search engines then your Website or Blog should have a proper robots.txt file. I hope now you have enough informaition about What is robots.txt file and how much important it is for your website or blog but you have any problem regarding this feel free to ask me via your comments. If you feel this information is useful then do not forget to share it.

Thankyou for giving your time.