Why is the crawling process looping indefinitely?
Description
There have been reported cases in which the crawling process never seems to end. A user might notice that the vulnerability test execution never begins, and that the crawling stage keeps looping indefinitely.
Cause
This can happen for many different reasons, many of which are related to an unknown behavior depending on the way the dynamic site was constructed. It might also occur that a web server terminates the connection with the scanner due to the suspicious number of consequent requests it sends out.
Resolution
If this situation is occurring, follow these steps:
- In the tools explorer, select the 'Site Crawler' tool.
- Enter the target URL and click 'Start'.
- When you notice that the site structure stopped expanding (usually after 10 minutes, depending on the size of the site) click the 'Stop' button to terminate the crawler process.
- Next, right-click the 'Site Crawler' entry in the Tools Explorer
- Choose the option to save the crawl results as a cwl file
- Click the 'New Scan' button in the top-left corner of the product, and the scan wizard will appear.
- Instead of entering the URL to be scanned, select the 'Scan using saved crawl results' option, and load the previously saved cwl file
- Continue the wizard process till the scan is started. This will ensure that the crawler will us the previously constructed site structure, and no looping will occur.
|
