When used by a website, URL rewrite rules need to be defined in Acunetix WVS to instruct the Crawler on how to recognize rewritten URLs, otherwise some URLs will be misinterpreted as directories — which will result in an incorrect scan.
In this FAQ we will detail one of the URL rewrite rules needed to scan the sample site http://testphp.vulnweb.com/Mod_Rewrite_Shop/, where /Mod_Rewrite_Shop/ uses URL rewriting technology. Three rewrite rules are needed to scan this website; you can download the rewrite file to follow this exercise.
The below URL will request the web application to display product details for the product: ID 1.
The above URL, is rewritten by the web server. In fact, when the web server receives the request, it sends the following URL to the web application;
From the above, we can see that details.php has changed to /Details/ (looks like a directory), the product name is displayed as a directory, and id=1 has been changed to 1.
- To set the above URL rewrite rule in Acunetix WVS, click on ‘Add Ruleset’ from the Configuration > Scan Settings > Crawling Options > URL rewrite node.
- Click on the HostName to enter the website or web application URL.
- Click on the ‘Plus’ sign button to create a new rule.
- Since URL rewrite is only enabled for /Mod_Rewrite_Shop/ directory, select ‘Directory rule’ and specify the directory ‘/Mod_Rewrite_Shop/’. If URL rewrite technology is enabled throughout the whole website, select ‘General Rule’.
- In the ‘Regular Expression’ input field, specify a part of the URL including regular expressions (or a group of Regular expressions) which Acunetix WVS will use to recognize a rewritten URL. E.g. Details/.*/(d+). This means match everything after the Details/ directory, and after that matched string, match also a digit or more.
- In the ‘Replace with’ input field, specify the URL Acunetix WVS should request instead of the rewritten URL. E.g. /Mod_Rewrite_Shop/details.php?id=$1. The $1 will be replaced with the value retrieved from the first regular expression group specified in the ‘Regular Expression’ input field, in this case (d+).
Thanks to the above rule, when the Acunetix WVS crawler matches the following URL; /Mod_Rewrite_Shop/Details/network-storage-d-link-dns-313-enclosure-1-x-sata/1, instead it will request the following; /Mod_Rewrite_Shop/details.php?id=1.
Therefore in the ‘Regular expression’ input field, one must always specify a string including a group of regular expression that will match the dynamic URL. In the ‘Replace with’ input field, one should specify the real URL Acunetix WVS should request, including $1, $2 etc which will be replaced with the content matched in the grouped regular expressions.
For more information about regular expressions, click here (http://en.wikipedia.org/wiki/Regular_expression)
Get the latest content on web security
in your inbox each week.