Walt
posted this on August 15, 2011 10:26
We apply a standard (recommended) robots.txt file to our sites e.g. mysite.com/robots.txt
The content of the robots.txt file requests that bots (search engine robots that explore and read each website) ignore two of our directories: the "_app" and "cgi-bin" directories.
Specifically the robots.txt file contains the following:
User-agent: *
Disallow: /_app/
Disallow: /cgi-bin/
We are uncertain as to why they sometimes ignore the request and go and index these directories.
In our experience this does not negatively impact SEO even though you may get some advisory warnings from Google - who themselves will suggest that this may not be an issue. But generally this situation does self correct in a short period of time.