What is Robots txt and how to create Robots txt file? – 2018

0
what-is-robots-txt-file

Hello Tweakerlinkans, Today we will learn about what is Robots txt file and how to create robots txt file? When you go to a new place to see the place, there is a guide which tells you where you can go to the place and where you can not go. Now, robots txt also work like the guide which tells google bot where they can crawl and where they can’t.

What is robots txt file?

Robots.txt is like a simple notepad’s text file, but for a blog or website, it is very important because when any search engine bots come to your website or blog, then they first crawl robots txt file for website crawl path. the robots txt file tells the search engine about which web pages and folders are they can crawl and not in the website.

Why we use Robots txt file?

You may have known that by reading above what is Robots txt. now you are thinking that why we prevent them to index our website pages or file in google search engine by crawling. then let me tell you every website has some sensitive data, some under construction pages or folder which also contains sensitive pieces of information that website owner do not want to share with anyone, then they prevent search engines from searching those pages and files through a robots txt file.

How to create Robots txt file?

The robots txt file is created in your website’s root folder [for ex: yourwebsite.com/robot.txt]. If you want to edit it, you can use any text editor, let’s know about its commands …

The basic syntax for the robots txt file

User-agent: [The name of the robot for which you are writing these rules]

Disallow: [page, folder or path where you want to hide]

Allow: [page, folder or path where you want to unhide]

Example 1. If you want to allow crawl everything, then use this code (All search Engine)

User-agent: *

Disallow:

Example 2. if you want to Disallow to crawl everything (All search Engine)

User-agent: *

Disallow: /

Example 3. If you want to Disallow for the specific folder (All search Engine)

User-agent: *

Disallow: /folder name/

Example 4. If you want to Disallow for the specific file (All search Engine)

User-agent: *

Disallow: /filename.html

Example 5. If you want to Disallow for a folder but allow the crawling of one file in that folder (All search Engine)

User-agent: *

Disallow: /folderxyz/

Allow: /folderxyz/anyfile.html

Example 6. Allow only one specific robot access in website

User-agent: *

Disallow: /

User-agent: Googlebot

Disallow:

Example 7. To exclude a single robot

User-agent: BadBotName

Disallow: /

Example 8. If you want to allow for the sitemap file crawling.

User-agent: *

Sitemap: http://www.yourwebsite.com/sitemap.xml

Example 9: WordPress Robots txt File

User-agent: *

Disallow: /feed/

Disallow: /trackback/

Disallow: /wp-admin/

Disallow: /wp-content/

Disallow: /wp-includes/

Disallow: /readme.html

Disallow: /xmlrpc.php

Allow: /wp-content/uploads/

Sitemap: http://www.yourwebsite.com/sitemap.xml

Tip – Do not disallow files in the robots txt file that you want to crawl or hide, because by doing so you are telling everyone about those files, so putting them inside a folder Hide that folder

Extra Information – Do You Know What is Good Bots and Bad Bots?

Such software that is created for automated tasks on the internet, we call them bots. More than half of websites traffic come through bots but bots are also of 2 types good bots and bad bots, let’s know about them.What is Robots txt and how to create Robots txt file?

Good Robot Bots – These Robot bots are very helpful for your websites and users, they crawl the pages of your websites so that the good search engine results page can be displayed in addition to these helps in website updates, changing the price of a product, and find to copyrighted content. according to research, 36% of the web traffic comes through these bots. Good bots always follow the rules of the present robots txt file on your website, Googlebot, Bingbot, Slurp, and Msnbot are examples of good bots.

Bad Robot Bots – These Robot bots are very harmful to your websites and users, it steals data from websites such as financial data users, sensitive information, website admin login details etc. Bad robot bots can down your website server by consuming too much Bandwith. and it can completely damage your website.

so guys, I hope you liked “what is robots txt and how to create robots txt file” ” post. If you have any questions or suggestions related to this post, then comment below and share this post with your friends.

Also read: How to get Wikipedia Backlinks within 15 Minutes in Genuine way

LEAVE A REPLY

Please enter your comment!
Please enter your name here