robots.txt

Read 3954 times
This file is missing...

Code: [Select]
2013/02/08 08:30:03 [error] 5795#0: *172499 open() "/usr/local/centovacast/web/robots.txt" failed (2: No such file or directory), client: , server: centovacast, request: "GET /robots.txt HTTP/1.0", host: "-------------------", referrer: "----------------------------/robots.txt"

Was there suppose to be a robots.txt file included?  Are we suppose to make our own ?
CrossFire-Hosting LLC.
Co-Owner
No, and no.  More information about robots.txt.
Well, I know what a robot.txt file is/does, as I have one for a couple sites of mine, but the reason I asked is that in the error log Centova Cast is trying to find one, and it doesn't, so it says its missing.

But if its not suppose to have one, why is it trying to find it?
CrossFire-Hosting LLC.
Co-Owner
The log records HTTP requests. Search engines make HTTP requests for robots.txt to see if it's there. If you visit http://yourdomain.com:2199/phabluggurbhh.txt you'll see an entry in the log stating that /phabluggurbhh.txt is not found too. Doesn't mean it's supposed to exist.