Fortify Software Inc. has released a report, entitled Misplaced Confidence in Application Penetration Testing, that details overconfidence in application penetration testing.

The report highlights that users of application penetration testing poorly understand how to gauge the effectiveness of their penetration tests and is comprised of two parts: a survey of security testers and an in-depth experiment to validate survey results. While the survey revealed high expectations of application penetration tests, the experiment showed that automated and manual tests often reached only 25 per cent of an application's security critical APIs, leaving large portions of the code untested. In addition, the tests failed to identify critical vulnerabilities within the parts of the application they did cover.

"This research suggests many testers don't quite understand the effectiveness of application black box testing and the role it has in the secure development lifecycle," said Jacob West, manager of Fortify's Security Research Group. "We found that many believe this form of testing - whether conducted with automated tests or manually by ethical hackers-to be a comprehensive approach to test applications. While it's an important stage in the lifecycle, it's going to miss many hard to reach portions of the application."

Application black box testing uses various inputs to probe an application while it is running in order to simulate attacks and identify potential vulnerabilities. For this report, a member of Fortify's Security Research Group conducted automated and manual application penetration tests on five common test applications. The researcher used two of the top three market-leading application penetration testing tools, as well as manual efforts. Fortify then used its own product, Fortify Tracer, and deployed it inside the test applications in order to generate detailed data on the effectiveness of these application penetration testing tools.

This study exposed a significant gap between the expectations of consumers of application penetration testing and the reality of the results when measured in a systematic and objective manner. The results showed that at best, one of the tools achieved 29 per cent coverage averaged across five applications. Knowing that most companies augment automated testing procedures with manual testing, the tester attempted to increase the coverage percentages by adding manual efforts. Although these results showed an average increase in coverage of 19 percent, they still missed more than half of the vulnerable APIs in the applications.

Furthermore, of the APIs that were examined, Fortify Tracer showed several instances where automated and manual efforts failed to uncover key vulnerabilities particularly SQL injection and cross-site scripting related vulnerabilities. In addition, Fortify Tracer discovered types of vulnerabilities that the black box testing approach is not set up to detect, such as a privacy violation, which indicates the application has written sensitive data to a log file.

"Log files are particularly vulnerable to attack by hackers who recognise it's an easy way to extract sensitive information from a system," West explained. "Application penetration tests will miss this type of vulnerability because they operate outside the application, and can't actually see the application writing the information to a log file."

At the onset of the report, Fortify surveyed dozens of penetration testers at various organisations to gauge their confidence in and expectations of the effectiveness of application penetration testing. More than 58 per cent of the respondents said the security tests they run are adequately testing their applications for vulnerabilities, (defined as reaching at least 61 per cent of the security related APIs in any given application). In fact, 46 per cent of the respondents estimated that their application penetration tests were able to reach at least 81 per cent of their application's security related APIs.

"Because of its black-box nature, application penetration testing involves a great amount of guess work. As such, any particular test may not be as effective as it could be," said Chenxi Wang, a principal analyst with Forrester Research. "A better approach for organisations is to combine black box testing with some level of white box knowledge. Fundamentally, this emulates an extremely powerful attacker who has code-level information. If you can defend against such an attacker, then you can defend any attacker with only a black box view. The ability to see inside the application and study what's going on during the testing can provide extremely useful guidance to testers. Organisations will see increased test efficiency and cost saving benefits."

"Without insight into an application's internal code structure, testers will continue to reach somewhere around 25 per cent of the application, leaving large segments of code exposed and vulnerable," West added. "As testers become more aware that application penetration testing has its limitations, and take advantage of tools and methods that enable them to really understand what's happening inside their applications as they are being tested, then we will see a new, more measurable and more effective style of application penetration testing."

[This issue has been well known for many, many years but deserves every exposure it can get. I'm surprised the tools tested only reach as little as 25 per cent of applications under test - that has not been my own experience when using this type of tool - but I have only used them on small programs, up to perhaps 100k lines of code.

However, this issue is enormously important because it is one of the underlying issues of the whole vulnerability problem, that code is incredibly buggy despite developers using black box tests on it. What needs to change is the software development process. It basically needs to be wrapped in a risk management envelope and taken through a series of stages identifying e.g. likely attacks vectors and introducing and controlling protection based on risks and priorities. You can automate a lot of this, and you can iteratively improve the quality of automatic tools by adding cases, but it is fundamentally not an automatic stand-alone process. It requires intelligence and expertise, and hence it is costly.

The legislative and legal systems should take steps to make sure it is even more costly not to carry out secure software development. --Ed].

Related links: (Open in a new window.)
www.fortifysoftware.com/products/tracer/

View printable version (opens in new window)
Back