Robots.txt WordPress Plugin - Peter Coughlin

Robots.txt WordPress Plugin

This page is about my Robots.txt WordPress plugin, which is another one of those plugins designed for lazy people like myself, who just want to set something and not have to worry about it again.

What the plugin does

When you enable the plugin, it intercepts requests for a robots.txt file, and creates a virtual one on the fly. By virtual, I mean there's no actual file created, but one is returned to the visitor, whether they are a real person or a search engine.

By default, the robots.txt plgin has Google's mediabot allowed, a bunch of spam-bots disallowed, and a few of the standard WordPress folders and files disallowed.

As a little useful extra, if the robots.txt plugin detects an existing sitemap.xml file, or if you are using my Sitemap plugin, it will add a reference to your sitemap into the robots.txt file. Apparently this helps with discovery of your sitemap.xml and subsequent indexing of your pages.

How to use the plugin

  1. Download the plugin ZIP file (see below)
  2. Unzip the archive
  3. Copy the "pc-robots-txt" folder and all its contents to your WordPress plugins directory (usually /wp-content/plugins/)
  4. Log in to your blog admin, click the Plugins menu and activate the PC Robots.txt plugin

You can change what appears in the virtual robots.txt file from the plugin admin page. Just log into your blog admin, and go to Settings and click on the Robots.txt menu link.

Let me know if it works for you.

Download Robots.txt Plugin
Tags: Robots.txt, WordPress, WordPress Plugins
Published under WordPress Plugins

2 Responses so far ↓

Richard Brown
Sep 5, 2008 at 12:15 pm

This is a truly elegant solution to a total PITA.

Top job!

Shawn
Sep 5, 2008 at 12:18 pm

Does a robots.txt file help a blog get re-indexed in Google, if you’ve been kicked out?

My site was delisted from Google, I suspect for syndicating headlines of everyone elses content with a link back to the full text article. Do you know how I can fix this?

Thanks kindly,
Shawn

Tell me what you think...

Note: your comments may need to be approved before they are shown.