You can tell that I am too busy when I take almost three months to blog about something interesting and useful to a wider audience. This is one of those occasions. Earlier this year Adrian Asher, Head of Security at Betfair, kindly invited me to visit their performance testing lab. Consequently in May I spent one day testing the performance of ModSecurity 1.9.x working in the reverse proxy mode.
When you are using ModSecurity embedded performance is usually measured in milliseconds, as the time needed for ModSecurity to do its work. The most relevant information here is the relative speed. If the application you are protecting spends over 100 milliseconds processing every dynamic request, a millisecond here or there won't make a difference. Those deploying ModSecurity in the reverse proxy mode, on the other hand, usually wish to know how many requests can be processed every second. They need to know if the reverse proxy setup is going to be fast enough for their needs.
Testing of a reverse proxy setup requires one to execute a large number of transactions per second, hopefully simulating real-life conditions. In addition to the reverse proxy box, you need to simulate a bunch of clients and a bunch of web servers. This can be problematic because, even with modest hardware, one cannot easily saturate the reverse proxy box using only one client machine. In addition, you typically need more than one web server machine. That's why I jumped at the opportunity to visit Betfair - they have specialised testing equipment, namely an Avalanche and Reflector pair. This (expensive) equipment will take care of the client and web servers part, allowing you to focus on the box that sits in the middle.
The reverse proxy box we used was a Sun v20Z machine (single AMD Opteron 244 running at 1.8 GHz, 1 GB RAM, Gigabit network) running Red Hat Enterprise Linux (kernel 2.6.9-11 EL). One day sounds like a lot of time but it really isn't for things like this. Although we worked hard throughout the day we only managed to test a dozen of different (software) configurations and no tweaking of the machine or the operating system was performed. You can see the results on the attached graph (red = Apache alone, yellow = ModSecurity with 25 rules, green = ModSecurity with approximately 180 rules - the entire certified rules set). So although it is certainly not comprehensive this test is probably close enough to what you would experience in a typical deployment. Some things to note:
- ModSecurity performance continues to not be a problem. (Unless you shoot yourself in the foot by deploying thousands of rules, that is.)
- The CPU was the bottleneck in all our tests. Hence simply increasing the CPU power would yield better results.
- The overhead of ModSecurity by itself is very low. Almost all of time ModSecurity uses is spent processing regular expressions, meaning you are in full control over the performance.
- Apache w/ModSecurity running with a generous rule set can push through at least 1500 requests per second with latency under one millisecond running on modest hardware.
One thing I noticed during my tests is that most of the rules are actually very simple patterns yet they are executed through the regular expression engine. One of the things I will do very soon is introduce other pattern matching algorithms to ModSecurity 2. That will probably help us increase the performance soon. More information coming soon!
