A study just published by the CyLab at Carnegie Mellon University shows that anti-phishing browser toolbars are generally not up to the task.
The study, Phinding Phish: An Evaluation of Anti-Phishing Toolbars by Lorrie Cranor, Serge Egelman, Jason Hong, and Yue Zhang, examined 10 of the 80-90 free anti-fraud toolbars currently available.
Initially, the researchers used a phishing site feed from the Anti-Phishing Working Group (APWG), which was fed into an automatic system, basically consisting of a series of bots, each testing a particular toolbar, controlled by a task manager which looked at the APWG feed. The Task Manager carried out some initial processing of the information in phishing messages to isolate and remove legitimate sites before testing the remaining URLs. In fact, the way this was done demonstrates some heuristics which are in themselves interesting and could be incorporated into an anti-phishing system. The result of the processing was that the researchers ended up with URLs to sites which had a high probability of being illegitimate.
The bots sent page requests to the sites thus identified, using an anonymiser to make it less likely that scammers noticed they were being observed. In each case the bots noticed the reaction of the toolbar under test and returned this information to the task manager.
Subsequently this method was refined and manually verified phishing sites obtained from phishtank.com used in the tests. Additionally, the toolbars were tested with 510 verified legitimate URLs in order to test for false positives.
The results were as follows:
| Product | At First | 24 Hours Later |
| Spoofguard | 91 per cent | 91 per cent |
| EarthLink | 83 per cent | 84 per cent |
| Netcraft | 77 per cent | 80 per cent |
| Google* | 70 per cent | 84 per cent |
| Cloudmark | 68 per cent | 67 per cent |
| IE7 | 68 per cent | 67 per cent |
| TrustWatch | 49 per cent | 51 per cent |
| Ebay | 28 per cent | 26 per cent |
| Netscape | 8 per cent | 21 per cent |
SpoofGuard, which uses only heuristics to determine whether a site is suspicious or not, seems to be doing very well until you look at the false positives. Unfortunately it also comes up first here, with no less than 38 per cent of the legitimate sites incorrectly identified as phishing sites, and 45 per cent 'unsure', making this a useless tool.
EarthLink suffers from similar issues, with only 1 per cent false positives but 91 per cent 'unsure'. Netcraft, Google (and by implication, Firefox 2.0) and IE7 displays no false positives and no 'unsure' results, but even the best of these, the Netcraft Anti-Phishing Toolbar, only identified 77 per cent of the malicious sites as malicious.
You can glean a bit more information from the table, such as how fast blacklists get updated (look at the 24 hours later, during white period of time some of the malicious sites were taken down and some blacklists updated). The original paper contains a lot more information than extracted for this article, including attacks against these toolbars (URL below). The researchers also included McAfee SiteAdvisor in their tests. It scored 0 per cent. I have left it out because it is not clear to me whether this tool is actually intended to be an anti-phishing tool.
There are a couple of remarks to make about the study, itself. Only two computers were used to run the bots, so there is a time factor which is not discussed. Judging from the tables in the report it is not significant. Furthermore, it is not explicitly discussed whether the responses of the toolbars under test in any way depend on the browser under which they are run. Since this point has been left out one can perhaps conclude that this is not the case.
It is safe to conclude, based on this research, that the performance of anti-fraud toolbars in general leaves a lot to be desired. None of them are very good, so these types of 'semantic' attacks still rely on their targets, humans, to discover and defeat them through safe browsing habits and suspicion against any information seemingly sent them by trusted correspondents such as banks. These toolbars provide no firm defence against fraud.
As an interesting aside: all the toolbars rely on a red/green indication scheme for site maliciousness. Considering that the prevailing type of colour blindness is red-green this may not the best choice, touching on one of your editor's favourite hobby-horses, that usability is (or should be) an integral part of security.
Related links: (Open in a new window.)
www.cylab.cmu.edu/files/cmucylab06018.pdf
View printable version (opens in new window)
Back