One of the most prevailing issues that tanks a website’s ranking in the SERPs and is flagged almost every time you conduct a Website Analysis is duplicate content.


A duplicate content scenario arises when the same content exists on more than one page where the same can mean true duplicate (100% similar) or near duplicate.

While these scenarios do not create a penalty (as is commonly believed), it is a good practice to tackle duplicate content to stand out from the rest.

The “duplicate content struggle” is more common than we know. So here are a few ways we have laid down for you to help you plan your killing spree for duplicate content:

NOTE: If you have trouble finding duplicate content, you can use Site Auditor to excavate duplicate content instantly along with other related on-site issues.


1. 301 redirects


Setting up a 301 redirect to the original page (from the duplicate page) eliminates the problem related to duplicate content.

What a 301 redirect does is, it tells the audience (both humans and bots) that there is a new permanent address for the page they are looking for. So the transition done by 301 is often smooth and seamless.

When doing these redirects, it is not just about choosing the preferred URLs; it is equally important to ensure the same syntax and URL parameters.


Tip: For Wordpress, the Redirection plugin can ease your process.

2. NoIndex metatag
NoIndex value in meta-robots instructs the search engine to not index that particular page while still allowing it to crawl (unless you instruct it not to).

There are variants for the meta robots tag. You can achieve this through the following two formats:

The first format is useful when you WANT the search engines to crawl your website while still telling them not to index it. The other format is useful when you want to restrict them from following the individual links on your page (again these links will be crawled but not indexed).

3. Blocking in robots.txt
If you want to hide your duplicate content from the search engines but still keep it visible for your audience, then robots.txt will help you do the trick.

A disallow command in the robots.txt file can be used to block entire folders easily. However, this practice is good for blocking only the duplicates that are not indexed. But, for content that is already indexed, the robots.txt method is not advisable.

4. rel=canonical
This tag gives you the page concealment powers which can help you shadow duplicate content. It tells the search engine which link to show when users perform a search for it.

Rel=canonical will take the user to the page’s canonical version regardless of the URL accessed to reach it.

Let’s understand the above with an example. When the user enters your site through, he will simply be taken to the canonical version, which is

5. 404 not found
To kill duplicate content, you can either hide it or remove it altogether. But removing something which is of value, isn't a very good idea.

So, this technique can work well if you want to kill duplicate content that doesn't have any good backlinks or isn't significant in terms of authority.

This means that every time a user tries to access this particular page, he simply gets a '404 not found' error.


6. Hashtag instead of question mark in UTM parameters
For URLs that contain a '?' operator, the search engines crawl these websites and can report instances of duplicate content.

Replacing the '?' operator with a hashtag (#) helps in addressing the issue.

Everything that comes after # sign in the URL is ignored. So when search engines do come across these in your links, duplication gets eliminated.

7. Minimise similarity while creating content
A good practice to get rid of problems related to duplication is not to create them in the first place.

Though it is an "unachievable scenario" for most of us, it is beneficial to focus on creating content that is not a mirror image of the content on some other page. (in case you do have mirror images, there is rel=canonical for your rescue).


Here is what Google suggests:

"if you have a travel site with separate pages for two cities, but the same information on both pages, you could either merge the pages into one page about both cities or you could expand each page to contain unique content about each city."

● Use 301 redirects to take your audience to the new permanent address for the content they are searching.
● Robots.txt can help in addressing the duplicate issues but its use is only advisable when the page in question isn’t already indexed.
● Choose the canonical version carefully.
● Follow good content practices from the beginning to prevent these issues from arising.

Even though search engines do not penalize you for using duplicate content but they do frown upon such issues. Therefore, dealing with duplicate content should be your priority.

Using deceptive practices to cover up content problems can remove you from the search results.

The above few practices can help you tackle duplicate content in the right manner. So put them in your content checklist right away!

Author's Bio: 

Information Transformation Services(ITS) is an IT and back-office support services company. ITS offers a comprehensive range of business process outsourcing solutions, tailor-made for each customer. With years of experience servicing a diverse range of industry leaders around the globe, ITS has developed its staff and facilities to meet the requirements of any data or resources intensive projects.