Prevent profile indexing on search engines

Hi everyone,

I want to prevent my users' profiles from being indexed on search engines.  Where should I place the robot.txt to prevent this?

Thanks in advance for the help!

Hi,

I have sort of the same question.

My community has 3 parts.  The front end is made up of html pages, there is a wordpress blog attached, and the dolphin community on the backend.  I want the blog and front end to get indexed but not any of the dolphin community.  I want to keep all bots away from there.  Is this possible?

I read that robots.txt is counter productive so I'm confused on how to set up the .htaccess just for the dolphin part.

Simply set access to members only for profiles & profile links.  This will cause the server to serve up a denied page to the bot.

Oh great! Thanks!

Not sure how to do this. Could you explain how this is done?

Thanks

You can do this through membership level permissions, which you'll find under Administration > Settings > Membership Levels. Click on the membership level you want to edit and activate or deactivate certain permissions. You can further control permissions by clicking on them, which will bring up a new window with more options.

You could also try this if your want the profiles to remain viewable by those visting the site but prevent the search engines from indexing them

It can be done by inserting a injection into the head of the profile page with the meta tag of <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">


To do this run the following query in phpMyAdmin.

INSERT INTO `sys_injections` (`name`, `page_index`, `key`, `type`, `data`, `replace`, `active`) VALUES ('profile_noindex', 5, 'injection_head', 'text', '<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">', 0, 1);

Then clear your dolphin cache.


You really can't do this via the robots.txt file because profiles resolve to anything after the domain for anything normally not found. Kind like a 404 page not found pretty much attempts to load a profile page instead. Difficult to explain.

To do it via the robots.txt would require you specifically allow access to all known scripts you want indexed and deny everything else. The injection i suggested above would be far easier.

deano,

Very helpful, thanks for the response. From what you're saying, sounds link including profile.php in robots.txt won't do the trick?

Can a file like browse.php be included in robots.txt to prevent indexing of profiles displayed in Browse Member blocks?

Your approach covers the above scenario, but I'm curious if it would work,

Thanks in advance!

Anything that normally appears in the url can be placed in the robots.txt file. The profiles can't becuse profiles.php does not normally appear in the url. It's rewritten by the .htaccess file.

So you can block browse.php and the robots will not follow the links on that page, but it will not prevent indexing of the profiles becuse links to the profiles appear on many other pages as well.