Robot Exclusion Protocol
Robot exclusion protocol is the standard used for robot exclusion files, which defines the syntax and location for robot.txt and direct web robots how to parse the file. Its file must be called robots.txt and it resides in the server’s document root.
Permalink Comments Back to Top Back to Homepage
