I'm wondering how I am going to figure out a site structure problem. The issue is that the search engine spiders are going to crawl http://DOMAIN.com/index.php which shows up as the guest area of the website (as opposed to http://www.DOMAIN.com which is the home ("splash") page. When the robots start following links in there, they are going to crawl thousands of pages that I don't want them to as they follow links to people's profiles, photo upload, account configuration pages, etc. The spiders will get lost and never make sense of the site. Is there any way to block this from happening or do a 301 redirect to the landing (splash) page which I want to serve as the home page? Should we have installed Dolphin in a sub-directory and created the splash page as "index.php" in the root directory? Would that work? I want the spiders to crawl the home page (splash page) and follow the links to my blog, articles and other content. Is there any way to block the spiders off from the member's (guest's) area? Note: I would prefer blocking off guest access completely and not having any guests. |
You can do some changes in google web master tools. ---- |
anything I can do on the site itself? robots.txt? .htaccess? although Google is the most important (especially internationally), I would also like to control the Yahoo/Bing bots |
You can use robots.txt to deny access to unwanted urls. ---- |
Is there any way for me to block guest access completely and just have domain.com/index.php forward to a certain page (unless the user is logged in). I don't want guests on my site at all. |