http://securosis.com/blog/death-of-product-reviews

We do recommend the same procedure; you should always try the product against the website or web application it will be scanning. If you are interested in a trial version of Acunetix WVS, contact our sales on sales@acunetix.com.

HP’s response to their results in Larry’s report was to encourage us to run WebInspect ourselves and make our own conclusions and not to make accusations.

Please pay attention to the conversation by reading what others have to say. It would also be great if you think before you post a comment.

Honestly, the way that you describe Point-and-Shoot is basically equivalent to seeing if the website owner has malformed HTML. Without logging into a web application? What are you thinking?

@ jericho: it’s a DSL-E prefix according to ARIN, RADB, and live BGP

1. Is the log entry timestamp meaningful? Is that when Suto was conducting his tests? If not, was it ‘coincidence’ that NTO had a developer testing a competitor’s web site?

2. Dan Kuykendall’s mock outrage is actually very telling. Read between his lines. Are you honestly NOT that up on your competitors? Are you not using their utilities to determine their capability? If your developer was running this scan, why from a “dsl extreme” IP and not from an NTO owned IP to be clear what was going on? If an NTO dev was running your product against a competitor, I think it is safe to assume it is so that your product performs well against a competitor demo app, specifically for the purpose of selling your product. If not, you can install a handful of 3+ year old PHP apps and get better diagnostics and tuning data. Last, a lot of “we” that is implied, but why not sign in a manner that is clear you work for NTO in some capacity?

3. Suto, why not publish all the results of your testing, including timestamps / reports of the scanners. Let people see the raw data to know it wasn’t you using a special copy of NTO. That would put one issue to rest, yes?

However, what you did was not Point and Shoot. That’s where we disagree. For some scanners you entered the credentials and for our scanner you didn’t. If it’s Point and Shoot then let it be Point and Shoot: just enter the URL, hit enter and let the scanner do his job.

Having the vendor expertly tune the site beforehand it’s not good either because some vendors might go too far and do more than tunning.

As a conclusion, I think that the best comparison would be to take a list of real open source web applications (so everybody can reproduce the results) on different platforms (PHP, .NET, Java, …) and scan them using Point and Shoot. The scanner that finds the most vulnerabilities wins. The vendors should be informed after the results are completed and not before.

I think you bring up an issue that needs to be addressed in further comparisons…the point and shoot category seems to cause the most controversy…maybe for future tests the vendor can supply a point and shoot config or something. And if you look at HPs recent response they seem to indicate that these sites are unrepresentative of typical applications as they create unusual security situations that are designed to show off scanner capabilities….so are we saying point and shoot is a meaningless category and the only fair test is to have the vendor expertly tune the site beforehand?

Larry

Yes, Larry told me that he didn’t recorded a Login Sequence for our test website. Therefore our scanner had to find vulnerabilities in an authenticated area without any credentials. We don’t have automated form based login support because I don’t see the point in that. The application can log you out anytime so I don’t see much gain from implementing that. Our Login Sequence Recorder is much more flexible because it automatically detects when the session is invalidated and reruns the login sequence.

@Larry Suto: By trying to keep it consistent you made it unfair for us.

Someone brought this to my attention. I just want to reiterate that many of the vendors new about the study and nothing prevented them from researching the sites.

Also point and shoot did not include any login macros…many of the scanners work with just entering username and password…I tried to keep it consistent in that way

I also asked Larry about the vulns on you missed on your test site, and he said that your tool does not have automated form login support, and that users are required to generate a Login Macro. So when he did Point and Shoot, it meant just that. His Trained scan was the one that he did macros for, and this was consistent for all the scanners. As far as I understand, all the other scanners have automated form based login support, including ours.

This post was mentioned on Twitter by devteev: @acunetix Acunetix’s feedback about “Accuracy and Time Costs of Web Application Security Scanner Report” by Larry Suto, http://bit.ly/8Yc2Jz...

I agree, Something smells bad…

LOL %))