URL Rewrite Rules

Web application developers use URL Rewrite Rules to hide parameters in the URL path structure. This makes it easier for search engines to index all the pages on a website, web browsers are supplied the URL in a format they understand and makes them easy for users to remember.

  • For example, when you browse a hardware store online library, the URL typically looks something like this: http://www.example.com/tools/hammer/. Using a URL rewrite rule, the web server converts this URL to a specific format, so it can retrieve the data from the back end database and display the tool details to the website visitor: http://www.example.com/library.php?tools=hammer.
  • In this example, we can determine that the subdirectory ('/tools') in the first URL is actually a parameter in the library.php file that accepts inputs, which in this case is the tool name ('hammer').

During the scan, Acunetix 360 sends normal HTTP requests to the web application to mimic an attacker, ensure that such requests are accepted by the web application, and all parameters in the URLs are properly scanned. It is also possible to scan pages which have more than one parameter in the URL.

For further information on how URL Rewrite Rules work in Invicti, see How Invicti handles URL rewriting.

Problems with URL Rewrite Rules

This table lists and explains the problems that can occur when automated web vulnerability scanners attempt to scan websites that employ URL rewrite technology and rules.

Problem

Description

Parameters in URLs are not scanned

A common problem web vulnerability scanners have when scanning web applications that use URL rewriting technology is that scanners are unable to identify parameters in the URLs. Scanners assume that the URLs are directories rather than parameter names or values, and leave them unscanned.

For example, when scanning the URL http://www.example.com/tools/hammer/ the scanner would think that both tools and hammer are directories, while in reality tools is a parameter and hammer is a value.

Prolonged vulnerability scans

This problem can lead to prolonged scans and incorrect scan results. For example, if the web vulnerability scanner is scanning a tool database that contains 100,000 tools, since the scanner is unable to identify that there is a parameter and a value in the URL, it would think that they are all different pages, so it will try to crawl them and scan them all.

If memory problems and other exceptions are not handled properly by your scanner, this could also lead to your software crashing, leaving you with no results and a number of wasted hours.

If you do not configure URL rewrite rules in Acunetix 360, it will heuristically identify the pattern and will limit the scan to avoid having prolonged scans and incorrect results.

Configuring URL rewrite rules is a difficult process

Since URL rewrite technology has become really popular in web applications, many commercial web vulnerability scanners allow users to configure the scanner so it can identify the parameters in the URLs and scan them.

But even though web vulnerability scanners can be configured to scan websites using URL Rewrite Rules, there are several other problems users can face:

  • Configuring of URL rewrite rules support is very difficult
  • User must know how to write regular expressions
  • User should have access to web server configuration files

Unless you are the developer of the web application itself or have a deep understanding of the web application, and unless you have direct access to the configuration files, it is impossible to configure URL rewrite rules on the scanner. And, even if you do know how to do it, configuring rewrite rules is a very difficult and time consuming task.

Web applications are not properly scanned for vulnerabilities

Assuming you manage to configure URL rewrite rules in your web vulnerability scanner, there are further problems, or at least, there are a number of limitations to how the scanners scan the web application.

As a security precaution, web applications do not accept HTTP requests which are already ‘translated’, such as http://www.example.com/library.php?tools=hammer. By default .NET web applications do not accept such HTTP requests. The problem is worsened when scanning MVC web applications because such applications use a different approach to URL rewriting.

While Acunetix 360 can scan MVC web applications, many other web vulnerability scanners cannot, even when URL rewrite rules are configured.

Once you configure the URL rewrite rules in your scanner, it sends a type of HTTP requests called translated queries. Even though the web application security scanner reports that the scan ran successfully, most of the HTTP requests were denied and the parameters in URLs were not scanned, providing you with a false sense of security.

Information

By default, Acunetix 360 can heuristically identify URL rewrites in the target website.

However, if you can configure the URL rewrite rules manually as explained here, the scan will be more efficient.

Specifying the Parameter Type

Specifying the correct parameter type makes the web application security scan more accurate.

  • For example, imagine specifying the following pattern: "/{PARAM}/{ID}", where "{PARAM}" is the parameter that will be scanned and "{ID}" is its value. If the parameter type is not specified, leaving the default 'Any', this means that both the below URLs will match this URL Rewrite configuration:
  • http://www.example.com/products/18
  • http://www.example.com/products/date.js
  • The above matching is too generic and might lead to incorrect scan results. For example, Acunetix 360 might not scan some files because after collecting enough samples (i.e. /product/1, /product/2, /product/3 etc) it will assume that /products/date.js is just another value of the product parameter, due to the generic matching.
  • In this case, if you set the parameter type to 'Integer', you avoid this problem, since Acunetix 360 will only expect integers when scanning such parameters and when it notices something else, such as /product/date.js it will recognize it is a file.

Encoded URLs

If you are manually configuring URL Rewrite Rules, and your website URLs use encoded values, always specify the decoded value.

  • For example, suppose the Website URL is:

http://www.example.com/user/john%2dDoe

  • The correct URL rewrite rule should contain the decoded character as follows:

http://www.example.com/user/{firstname}-{lastname}

  • This rewrite rule is incorrect because it contains the encoded character:

http://www.example.com/user/{firstname}%2d{lastname}

URL Rewrite Fields

This table lists and describes the fields in the URL Rewrite tab.

Field

Description

Root Path Max Dynamic Signatures

If a URL block in the root path contains more items than this limit, it will be identified as a URL rewrite parameter. It must be between 1 and 10,000.

This field is displayed only in the Heuristic tab.

Sub Path Dynamic Signatures

If a URL block in the sub path contains more items than this limit, it will be identified as a URL rewrite parameter. It must be between 1 and 10,000.

This field is displayed only in the Heuristic tab.

Block Separators

Enter separators to use to split the URL into blocks.

This field is displayed only in the Heuristic tab.

Analyzable Extensions

If the URL contains a file extension, it will be analyzed only if the respective extension is in this list.

This field is displayed only in the Heuristic tab.

Enable Heuristic URL Rewrite detection

Acunetix 360 will try to automatically detect other URL rewrite rules if this option is set.

This field is displayed only in the Custom tab.

Placeholder Pattern

This contains the relative path with placeholders for URL rewrite parameters.

This field is displayed only in the Custom tab.

RegEx Pattern

This is a regular expression used for matching the URL rewrite parameters.

This field is displayed only in the Custom tab.

How to Configure URL Rewrite Rules in Acunetix 360
  1. From the main menu, select Scans, then New Scan. The New Scan window is displayed.
  2. In the Scan Options section, select the URL Rewrite tab.
  3. Select an option: None, Heuristic or Custom:
  • If None is selected, then no rules will be applied

  • If you keep Heuristic as the default, then:
  • Acunetix 360 will automatically populate the fields:
  • Root Path Maximum Dynamic Signatures
  • Sub Path Maximum Dynamic Signatures
  • Block Separators
  • Analyzable Extensions

  • If you select Custom:
  • Select Enable Heuristic URL Rewrite Detection so Acunetix 360 will attempt to automatically determine other URL Rewrite rules. Both Custom rules and Heuristic rules will be in effect. Or disable, so that only the Custom rules will be in effect.

  • In the Placeholder Pattern and RegEx Pattern fields, enter the relevant information
  • Click New to enter further rules
  1. Click Launch.

 

« Back to the Acunetix Support Page