Webmasters seeking to maximize search engine optimization (SEO) occasionally run afoul of Google search rules and are penalized for duplicate content. As explained in our October 7 post, the problem is one of differing goals: ranking vs. filtering. Webmasters who want site content to be accessed by as many search parameters as possible often assign multiple URLS to the same content. Working to produce unique search results, Google’s filtering process groups duplicate content and selects a representative site or page for display. Webmasters often feel they pay a “penalty” for unintentional duplicate content through decreased Internet exposure and display of less than optimal sites/pages.

Try these tips to minimize duplicate content and maximize your SEO exposure:

If you participate in an affiliate program, avoid the “cookie cutter” approach to site content. Your site should contain unique, relevant content that adds value to that found on the parent site.

Don’t use the “cut and paste” approach to creating content on multiple pages or sites. Some repetition of key phrases is unavoidable, but each page within a site should carry unique content with a distinct focus. For boilerplate text such as copyright info, use a short summary and link to a detail page.

Avoid scraping content from other websites; it’s a definite Google foul. However, there’s no penalty (besides annoyance) if someone scrapes your site. Google is quite good at determining which is the original site. If you syndicate your content, ask syndication partners to identify and link back to your content as the original source.

Use the robots.txt file to block less desirable printer versions of content from access by Google crawlers. Check the file to verify that content you want crawled hasn’t been blocked. If you have created multiple versions of your site, include the preferred version of your URL in the sitemap file. Google does not consider multiple language versions of a site to be duplicate content.

Use 301 redirects in the .htaccess file to redirect Google’s spiders.

Keep internal linking consistent.

Get rid of publishing stubs by either not publishing or blocking empty pages.

Use top level domains (TLD) to denote country-focused content.

Increase Visibility offers a free website analysis that can tell you how Google and other search engines see your site. Our trained SEO specialists can show you how to optimize your site and improve your search ranking.

Google’s “duplicate content penalty” gives web designers fits, despite Google’s steadfast denial of any “penalty.” It really doesn’t matter who’s right; it all boils down to a matter of semantics. Webmasters and Google simply view “duplicate content” from different perspectives: rankings vs. filtering. 

Duplicate content can occur when webmasters attempt to improve search engine rankings and increase traffic by utilizing popular search queries. Duplicate content should not be confused with plagiarism in which content is lifted from one site and republished as unique content on another site. That’s a definite Google foul, as is republishing content without adding additional value. Duplicate content is the non-malicious and usually unintentional repetition of chunks of content within or across domains. It generally occurs when pages are stored in multiple formats with multiple distinct URLs. It’s easy for multiple key words in a search engine to pull up the same content from different URLs, effectively “duplicating” the same content. The problem, besides the obvious desire to present unique content to searchers, is that searchers may receive a less desirable version of your webpage which, naturally, you want to avoid.

The disagreement over duplicate content occurs because website owners and Google don’t have the same goal. You are focused on ranking. You want your website or webpage to be among the first 10 to 20 pulled up in a search. Therefore, you want your unique content to be available from as many search avenues as possible. Google focuses on filtering. In a December 18, 2006 post on its Webmaster Central Blog, Google explained its approach this way:

“During our crawling and when serving search results, we try hard to index and show pages with distinct information. This filtering means, for instance, that if your site has articles in ‘regular’ and ‘printer’ versions … we’ll choose one version to list. In the rare cases in which we perceive that duplicate content may be shown with intent to manipulate our rankings and deceive our users, we’ll also make appropriate adjustments in the indexing and ranking of the sites involved. However, we prefer to focus on filtering rather than ranking adjustments.”

This is the origin of the perceived “duplicate content penalty.” Google says having multiple URLs on the same domain that point to the same content won’t earn you a trip to the principals office, but it does cause Google to react in a way that webmasters don’t find favorable, hence their sense of being penalized. Because Google’s goal is to show searchers 10 different results, Google groups URLs that point to duplicate content and selects what it considers the best URL to represent the group. Unfortunately, Google’s choice isn’t always the one you want searchers to see.

Friday: Avoiding Google’s duplicate content penalty