Indexing as many pages as possible on your website can be very tempting for marketers who are trying to strengthen their authority on search engines .
But, while it is true that posting more pages relevant to a particular keyword (assuming they are also of high quality) will improve your rankings for that keyword, sometimes it is more interesting to keep certain pages of your website off a search engine index.
... Say what?!
Stick with uss, guys. This article will explain why you might want to remove certain web pages from the SERPs (Search Engine Results Pages) and exactly how to do it.
De-indexing a page from Google
There are a few occasions when you might want to exclude a web page - or part of a web page - from search engine crawling and indexing like:
- To prevent duplicate content (when there are multiple versions of a page indexed by search engines, such as in a printable version of your content) to be indexed
- To process admin and pages connection for internal use unless they are intended for use by a community
- For a thank you page (i.e. the page a visitor arrives on after converting to one of your landing pages ) where the visitor has access to any offer from this promised destination page, like a link to a PDF ebook
Here's what the thank you page is for our SEO tips ebook looks like, for example:
You want toUs those who land on your thank you pages get there because they 've already filled out a form on a landing page - not because they found your thank you page in search.
Why not? Because anyone who finds your thank you page in search can go straight to your lead-generating offers - without having to provide their information to you to go through your lead capture form. Any marketing manager who includes the value of landing pages understands how important it is to capture these visitors as prospects first, before they can access your offers .
Summary: If your thank you pages are easily reperables through a simple Google search, you may be leaving some valuable leads on the table.
What's worse, you may even find that some of your top-ranking pages some of your long tail keywords could be your thank you pages - meaning you could invite hundreds of potential leads to bypass your lead capture forms. This is a pretty compelling reason why you would want to remove some of your web pages from the SERPs.
So how do you go about "deindexing " certain pages from search engines? Here are three ways to do it.
3 ways to deindex a web page from search engines
Robots.txt to De -Index
Use this if: you want more control over what you deindex and have the necessary technical resources.
One way to remove a page froms search engine results is adding a robots.txt file to your site. The advantage of using this method is that you can have more control over what you allow bots to index. The result? You can proactively exclude unwanted content from search results.
In a robots.txt file, you can specify whether you want to block bots from a single page, from an entire directory or even a single image or file. There is also an option to prevent crawling of your site while still allowing Google AdSense ads to work if you have them.
That being said, of the two options available to you, this requires the most technical kung. fu. To learn how to create a robots.txt file, you will need read this article which explains exactly how to do this. .
Customers: you can learn how to install robots.txt on your website here and learn how to customize the contents of the Robots.txt file here .
If you don 't need all the control of a robots.txt file and are looking for a simpler and less technical solution, then this second option is for you.
Htaccess No Index No Follow to De-Index
Use if: Your website is running on Apache and mod_headers is enabled, that's a quick fix.
In this case, you can attach this single line to your .htaccess file:
X-Robots-Tag header ", nofollow "
To signify that your website can be indexed, but never displayed in Google search results.
Meta No Index No Follow to De-Index
Use if : you want a simpler solution to deindex an entire web page and / or deindex links on an entire web page.
Use a tag to prevent a page from appearing in the SERPs - and / or links on a page - is both simple and effective. It only requires a tiny bit of technical know-how - in fact, it's really just a copy paste job if you use the correct content management system .
The tags that allow you to do these things are called " and "nofollow. " Before I get into how to add these tags, let's take a moment to define and distinguish the two. They are, after all, two completely different directives - and they can be used on their own or side by side.
Qu 'is a "no index " tag?
When you add a " tag to a web page, it tells the engine to search that although it can crawl the page, it cannot add the page to its search index.
Thus, any page containing the directive " will not enter the search engine 's index and therefore cannot be displayed in the results pages search engines.
What is a "nofollow " tag?
When adding a "nofollow " tag to a page Web, it prevents search engines from crawling the links on this page. This also means that any page ranking authority on the SERPs will not be passed to the pages it is linked to.
So any page with a "nofollow " " on it will have all of its links ignored by Google and other search engines.
As I said before, you can add a " directive either alone or with a "nofollow "directive. You can also add a single " nofollow "directive.
When to use " "and " nofollow "separately
Only add a " tag when you don't want a search engine to index your webpage in search, but you want it to follow them. s linkson this page - thus giving ranking authority to other pages your page links to.
Paid landing pages are a prime example. You don't want search engines to index the landing pages people are supposed to pay to see, but you might want the pages they link to have its authority.
Only add a "nofollow tag when you want a search engine to index your web page in search, but you don't want it to follow links on that page.
There aren't many examples of adding a "nofollow " tag to an entire page without also adding a " tag. When determining what to do on a given page, it's more a question of whether or not to add your " tag with or without "nofollow " tag.
When to use ", nofollow " Set
Add both a " and "nofollow tag when you don't want search engines indexing a webpage in search, and you don't want it to follow them links on this page.
Thank you pages are a prime example of this situation. You also don't want search engines to index your thank you page, or for them to follow the link to your offer and start indexing that offer's content.
How to add a " "and / or a " nofollow "
Step 1: Copy one of the following tags.
For " " ":
For "nofollow ":
For " "and " nofollow ":
Step 2: Add the tag to the HTML section of your page, aka the page header.
If you are a customer, it's very simple: click here or scroll down for instructions specific to Users of.
If you are not a customer, then you You will have to manually paste this tag into the code of your web page. Don't worry, it's pretty straightforward. Here's how to do it.
First, open the source code of the web page you are trying to deindex. Then paste the entire tag into a new line in the HTML section of your page, known as the page header. The screenshots below will guide you.
The tag means the start of your header:
Here is the tag for both " "and " nofollow "pasted in the header:
And the end of the header means:
Boom! That's all. This tag tells a search engine to turn around and leave, leaving the page out of search results.
No index, no follow in
Adding the " and "nofollow " tags is even easier. You just have to open the tool on the page where you want to add these tags and choose the "Settings " tab.
Then under Advanced options and click on "Head HTML. " In the window below, paste the appropriate code snippet. In the example below, I added both a " tag and a "nofollow " tag because it 's a thank you - your page.
Hit " Save, "and you are in gold.
No index successfully without following a page
You have magically deleted your results pagesearch engines. Now you can start capturing more of those lost leads again.
Now keep in mind that you won't see results instantly. Your changes will only be applied the next time your page is crawled by a search engine. Depending on how often you typically post new pages to your website, this may take a few weeks. The more frequently you post content, the more search engines will crawl your site. The best way to track how often Google visits your website is to review your exploration statistics in Google Webmaster Tools .
Bottom: If you notice your page appears still in Google search results, even with the " tag, it's probably because Google hasn't crawled your site since you added the tag. You can ask Google to re-explore your page using the Get as Google Tool .
Also note that some search engine crawlers may interpret these directives differently from Google le done, so your page may still show up in other search engine results. But for Google it will work fine, once you are done crawling your website.
Either way, you will be able to sleep a little easier knowing that you arehave finally created your website. a better place for your marketing.
Editor's note: This article was originally published in July 2016 and has been updated for completeness.
<--Call to action code ->