Limit Crawling to Address and Subdirectories Only
This option is useful to limit the scope of the scan to part of the web application. By default, the option “Limit Crawling to address and sub-directories only” is enabled for new targets.
This option will limit the scope of the scan up to the last forward slash (/) in the target address.
Note that any target URL WITH a path but WITHOUT a trailing slash will cause the crawler to consider the final part of the path to be a FILE and not a FOLDER; the result is that the parent folder of that file will be the real target URL. For example:
- the target URL http://www.example.com/folder1/subfolder1/ WITH the option "Limit Crawling to address and sub-directories only" will scan items beneath /folder1/subfolder1/ (WITHOUT the option, you will be scanning the full domain)
- the target URL http://www.example.com/folder1/subfolder1 WITH the option "Limit Crawling to address and sub-directories only" will scan items beneath /folder1/
🔍 Limiting Scan Scope - Examples
Therefore, if your target URL is set to http://www.example.com/task/subtask, you can disable the option "Limit Crawling to address and sub-directories only" to instruct the crawler to also look for resources in http://www.example.com/task/ and http://www.example.com.