Centova Technologies Forum

Centova Cast v3 => Bugs and issues => Topic started by: DJFire_CFR on February 08, 2013, 08:52:38 am

Title: robots.txt
Post by: DJFire_CFR on February 08, 2013, 08:52:38 am
This file is missing...

Code: [Select]
2013/02/08 08:30:03 [error] 5795#0: *172499 open() "/usr/local/centovacast/web/robots.txt" failed (2: No such file or directory), client: , server: centovacast, request: "GET /robots.txt HTTP/1.0", host: "-------------------", referrer: "----------------------------/robots.txt"

Was there suppose to be a robots.txt file included?  Are we suppose to make our own ?
Title: Re: robots.txt
Post by: Centova - Steve B. on February 08, 2013, 01:15:35 pm
No, and no.  More information about robots.txt (http://en.wikipedia.org/wiki/Robots_exclusion_standard).
Title: Re: robots.txt
Post by: DJFire_CFR on February 12, 2013, 07:27:14 am
Well, I know what a robot.txt file is/does, as I have one for a couple sites of mine, but the reason I asked is that in the error log Centova Cast is trying to find one, and it doesn't, so it says its missing.

But if its not suppose to have one, why is it trying to find it?
Title: Re: robots.txt
Post by: Centova - Steve B. on February 12, 2013, 07:43:24 pm
The log records HTTP requests. Search engines make HTTP requests for robots.txt to see if it's there. If you visit http://yourdomain.com:2199/phabluggurbhh.txt you'll see an entry in the log stating that /phabluggurbhh.txt is not found too. Doesn't mean it's supposed to exist.