If only Web application security were black and white. We could simply load our scanner without thinking anything through, enter the URL, click Scan, generate a report of issues for someone else to address and be done with it. Sadly I think some people do go about Web security this way but that’s for another discussion. The point I want to make here is that you cannot simply scan your websites and applications using a “best practices” scan policy and expect to find everything that needs to be discovered if the scan is indeed not looking for everything.

Some scanners such as Acunetix Web Vulnerability Scanner do a great job at checking for everything out of the box. Others will make you think you’re getting an in-depth scan such as OWASP Top 10 or similar best practices according to the software engineers who developed the scanner. However, if you look deeply enough you’ll likely realize that when running default or best practice scan policies the scanner isn’t going find everything that matters including flaws that should have been found otherwise.

I’ve seen this time again to the point where, in most situations, I’m configuring my scanners to just check for everything – to “have at it”. There’s too much at stake with that one big oversight you forgot to check for. After all, why wouldn’t certain XSS or SQL injection vulnerabilities show up when running an OWASP Top 10 scan? If you run this type of best practices scan and you’ll get one set of results. Run another scan checking for everything and you might get something entirely different. As crazy as it sounds I’ve seen this dozens of times. Pen tester beware.

No offense to any of the vendors. They’re doing the best they can. The responsibility to ensure all vulnerabilities are tested for lies in our own hands. This may mean using the default scan policy. It may be as simple as using a XSS policy. What is it that you’re trying to accomplish with your scans? Depending on your needs you may have to tweak any given scan policy to make it your own. When in doubt, have the scanner check for everything.

The important thing is to know the risks as there can be some involved depending on the scanner, how frail the system is that you’re scanning and so on. Denial of service comes to mind. To me, in most (not all) situations, the reward of finding an elusive vulnerability or three is much greater than the risks of taking the system down. Yet another reason to do your testing against staging, DR or other non-production environment.

The reality is that you absolute cannot say with near 100 percent certainty that you’ve checked for everything in your websites and applications if indeed you haven’t told your scanner to check for, well, everything. I think in many – likely most – situations we have the fiduciary responsibility to check for everything if that’s indeed what someone is expecting to be checked. So, plan ahead and proceed wisely. The payoffs can be huge.

Kevin Beaver

Kevin is an information security consultant with 30 years experience, providing independent security assessments and penetration tests, security consulting and virtual CISO services, writing and security content development, and speaking engagements keynotes, panel discussions, and webinars.

  • Shame. There is no scan policy that will find everything that matters… and there never will be.

    You set up a ridiculous strawman in order to support an irresponsible approach to appsec. Simply telling a tool to scan for “everything” is wrong in two key ways. First, tailoring a tool to an application is a lot more than flipping on all the scanner rules. You need to actually teach the tool how the app is supposed to work – login/logoff, access control, csrf approach, etc… Second and more importantly, even if you do teach the tool well, there are still plenty of issues that can’t be verified by tools.

    These tools are an important part of any appsec program, but this sort of positioning creates a dangerous false sense of security.

  • Thanks for the feedback Jeff. You’re right. I think we’re on the same page. There is no one best approach to testing *every* single application. There’s just too much complexity, customization and so on that’ll trip up even the best scanners and certainly help solidify a false sense of security.

    Based on my experience, the reality is if you’re going to run a vulnerability scan, why not scan for everything? I’ve just found *way* too many additional vulnerabilities by having the scanner run all checks to revert back to and rely upon some generic (and limited) scan policy the vendor thinks is best. Talk about a false sense of security!

    Ditto with manual analysis including tweaking the scanner’s additional settings, using its manual analysis tools for proxying, performing SQL injection, editing HTTP requests and so on. This stuff takes time…and patience. You have to plan ahead and determine what your ultimate goals are.

    You may be interested in some recent blog posts, articles and a sample book chapter of mine where I rail against canned approaches to security including the ignorant reliance on scanning tools alone:












    There’s only so much I can cover in a short and focused blog post. So, again thanks for bringing this stuff up.

  • Comments are closed.