Files listed in robots.txt but not linked

Description
  • This file is listed in robots.txt but is not linked anywhere in the site. A robots.txt file on a website will function as a request that specified robots ignore specified files or directories when crawling a site. This might be, for example, out of a preference for privacy from search engine results, or the belief that the content of the selected directories might be misleading or irrelevant to the categorization of the site as a whole, or out of a desire that an application only operate on certain data.
Remediation
  • In robots.txt you should only include files or directories linked on the site.
References