Comments on: Robots.txt WordPress Plugin

By: Peter

Peter — Mon, 08 Feb 2010 15:03:39 +0000

@Mark — if you don’t have a specific reason to use this plugin, i.e. to block spambots, then you don’t need to use it at all. But saying that, it won’t do any harm if you do use it..

By: Peter

Peter — Mon, 08 Feb 2010 15:00:43 +0000

You need custom permalinks for the plugin to work. When you have custom permalinks enabled wordpress will process request for non-existent files (including robots.txt) but if you don’t have custom permalinks enabled wordpress wil never even see the request.. you will just get a 404 not found error..

By: Mark

Mark — Fri, 05 Feb 2010 15:37:26 +0000

Peter, Google’s info on robots.txt says:

If you want search engines to index everything in your site, you don’t need a robots.txt file (not even an empty one).

So, if you’re activating the plug-in without making any amendments /blocking page/s will it harm your site in any way from Google’s point of view?

By: Ken

Ken — Sun, 24 Jan 2010 15:16:46 +0000

Installed easily, but when I click on -
“You can preview your robots.txt file”
(under “PC Robots.txt Settings”)
I get -
“Page Cannot be Found, Please try Search or Return to HomePage”.
Any advice greatly appreciated. (Concerned improper install, might hurt with SEs…
Thanks for the plugin! – Ken

By: MDMower

MDMower — Wed, 13 Jan 2010 19:52:30 +0000

I get the 404 error that a couple of others have mentioned, but I can’t seem to resolve the issue. Regardless of whether I manually put a robots.txt or remove robots.txt in my root directory (pagsa.missouri.edu/robots.txt), this plugin does not save changes to the file. I am using the “Google XML Sitemaps” plugin, and have tried using it with and without checking “Add sitemap URL to the virtual robots.txt file”.

By: Peter

Peter — Mon, 11 Jan 2010 18:58:44 +0000

Hi Peter, I have installed your plugin, but id doesn’t work, I only get 404 Not Found.
Oh, I use WP 2.8…:(

By: jack

jack — Wed, 09 Dec 2009 21:12:23 +0000

I am running the latest wp, with google sitemap, as the post last year I get webpage could not be found when I click on preview my file.
I am using the default permalinks, does that cause a problem. Deactivated it then reactivated it.

Any tips would be most appreciated.

Cheers

By: Peter

Peter — Tue, 01 Dec 2009 17:23:07 +0000

@Peter – please read the previous comments on this page – that question has been answered a few times already :-)

By: Peter

Peter — Tue, 01 Dec 2009 16:45:06 +0000

I installed your plugin and I am seeing the following among other code when I look at the robots file

User-agent: Googlebot
Disallow:

Does this mean to disallow google? If so, what do to allow google, msn, yahoo and small meta search engines. Thanks.

By: Peter

Peter — Tue, 01 Dec 2009 09:31:32 +0000

@Greg – Yes, you’ve got it right.. and thank you for the kind words..