Back in March 2006 I was approached by Forrester Research and invited to participate in their Q2 web application firewall evaluation, along with six other WAF vendors. I was delighted with their invitation and gladly accepted. It is not often that an open source product is invited to play with the commercial guys. It turned out the participation required a lot of work on my part. I had to systematically cover and describe the entire feature set of ModSecurity, and that's not something I do often (at least not with that level of detail). It was, however, a very productive exercise because I had to make a step back and look at a bigger picture.
The results were published a couple of weeks ago and I think we did rather well. We were praised for our positive aspects (e.g. everything is configurable) and criticised for our weaknesses (e.g. lack of a management GUI). Unfortunately the entire report is not available online - you would have to buy the report if you want to read it. Revealing excerpts are available for the main report and for ModSecurity.
Two quotes from the ModSecurity scorecard summary are of particular interest:
"...ModSecurity is by far the most extensively deployed Web application firewall, with more than 10,000 customers."
and:
"ModSecurity's stringent implementation standards — build nothing unless you approach the highest level of security — will push the entire Web application firewall market toward higher-quality products."
[Source: Forrester WaveTM: Web Application Firewalls, Q2 June 2006", Forrester Research, Inc., June 2006.]
P.S. Forrester are also making available a PowerPoint presentation that gives a quick overview of the reviewed products.
(August 7 Update) Michael Gavin, the lead researcher behind the Forrester WAF evaluation, wrote to me to say that their comments referred to the 1.9.x branch of ModSecurity, and that he expects ModSecurity 2.x to address most, if not all, of the issues they identified.