Is there any examples of well written robot.txt file dealing with dolphin? Please feel free to share here...
Robot.txt
 Csampson |
robot.txt only works for bots that follow the rules, like google and bing. If you are wanting to limit what sections that google and bing will craw, then yes, a robot file is the way to go. If you are thinking of bad bots they just laugh at robot.txt files. Geeks, making the world a better place |
robot.txt only works for bots that follow the rules, like google and bing. If you are wanting to limit what sections that google and bing will craw, then yes, a robot file is the way to go. If you are thinking of bad bots they just laugh at robot.txt files. I appreciate that I do...What I am looking for is to see how people use them. I get how they are used and why but just wanted to see if anyone had set one up that someone trying to learn might get something from it. My site has 500,000 pages and bots sometimes have issues but and however the new site maps seem to do what robot.txt use to do, I think? Csampson |
Below is the legacy version of the Boonex site, maintained for Dolphin.Pro 7.x support.
The new Dolphin solution is powered by UNA Community Management System.
The new Dolphin solution is powered by UNA Community Management System.