Advanced XML Sitemap Tips: 4 Pages to Exclude in a Robots TXT File

A Robots TXT file works together with an XML sitemap to communicate with search engines' robots, otherwise known as search engine spiders or bots. These are automatic scripts that travel the World Wide Web with the purpose of indexing content. While an XML sitemap tells the robots what web pages you want to have indexed, a Robots TXT file does the opposite, listing web content that you don't want robots to visit because you want to exclude them from search engines. Your Robots TXT file can disallow access to your entire website, certain folders or specific web pages. The file can be created using Notepad in Windows or TextEdit in Mac, and should be saved in the root of your domain.

There are 4 primary types of web content you would want to exclude from search engines using a Robots TXT file:

1. In-Progress Content

Imagine you're revamping a large website. When you peek into the website files, you find that it's filled with outdated web pages and worst yet, pages that bring up errors or redirect when people try to access them. These must be excluded from your XML sitemap and added to your Robots TXT file, so search engines won't accidentally index and send traffic to them. There's nothing more frustrating for a web user than thinking they've come upon the exact information they need, only to find that the page doesn't actually exist anymore. If you're working on your entire site for a while, you can even use the Robots TXT file to disallow access to your entire website, so search engine spiders don't visit your site at all.

2. Private Content

As a business, it's likely you have or are thinking of posting special content for people who pay or sign up. Likewise, you may password-protect parts of your website only for certain members of your team to see. Obviously, you don't want any of these pages showing up in search engines for all the world to see. By deleting these pages from your XML sitemap and inserting them into your Robots TXT file, you send a clear message to search engines that the pages are private and should not be displayed as public entry points to your website.

3. Low Priority Content

Considering how search engine spiders are automatic programs, you can't expect them to use common sense when they check out your website. They make their decisions based on calculations, so they don't consciously know what web pages you consider more important than others. For reasons ranging from traffic to exte
al links, a trivial page such as your privacy policy could be showing up at the top of search engine results for certain keywords. If you don't want certain pages to be your first impression to potential customers and clients, you should change their priority settings in your XML sitemap and then list them in your Robots TXT file.

4. Duplicate Content

Search engines penalize websites for duplicate content because spammers used to use this tactic to try and cheat the system. This means that you're in trouble if you offer your viewers a copy of the same web page for printing, use http secure (https) and less secure (normal http) versions of the same pages, or have different URLs that point to the same e-commerce store item for example. Make sure search engines knows that you're not deceiving them on purpose by clearly identifying the canonical (or preferred) page in your XML sitemap and listing any duplicates in your Robots TXT file for exclusion.