Sometimes we see ModSecurity users asking about performance in the mail-list. During this post I will talk about some important topics to improve ModSecurity performance.
1 – HTTP Caching and Acceleration
In a common web environment static contents (ie. Images) area substantial part of http traffic. Usually users don't want to execute ModSecurity rules against this kind of content. So the first recommendation is setup in front of ModSecurity a HTTP Cache and Acceleration solution.
We have interesting open sources solutions and one of them is Varnish. You can setup it in front of your ModSecurity and configure it to cache the static traffic. Once it is done, Varnish will start serving this kind of contents and ModSecurity will only see what is really necessary.
Another possible solution is setup rules to detect file extensions you want to inspect and ignore the others:
SecRuleEngine On
SecAction "id:'1', phase:1, t:none,setvar:'tx.inspect_extensions=.html/ .php/', nolog, pass"
SecRule REQUEST_BASENAME "\.(.*)$""chain,capture,allow,setvar:tx.exts=.%(tx.1}/,phase:1, t:none,t:urlDecodeUni, t:lowercase,id:2,logdata:'%{TX.0}'"
SecRule TX:EXTS "!@within %{tx.inspect_extensions}"
However this is not the best solution because even you will skip all other rules, ModSecurity will spend time buffering, forwarding and executing a few rules against this kind of data.
2 – Rule Selection
Rule selection is another important topic to talk about if you are using the OWASP Core Rule Set.
We have many categories of rules inside CRS, you should review them and decide if all categories and rules are important for you.
We recommend load all rules, however sometimes it is not possible for performance point of view. So you should do a risk analysis and load what is primordial.
3 – Rule Execution Mode
The rules from OWASP Core Rule Set Project can be executed in two different modes:
Self-Contained Mode - Rules inherit the "deny" disruptive action. The first rule that matches will block.
Collaborative Detection Mode - This is a "delayed blocking" mode of operation where each matching rule will inherit the" pass" action and will only contribute to anomaly scores.
From the performance point of view the Self-Contained Mode is the best solution since it should execute lesser rules than the Collaborative Mode, reducing the overhead caused by rule engine and logging engine. However from the False Positive point of view, Collaborative Mode should emit lesser False Positives.
That being said you should try the default mode first(Self-Contained) and decide if it is enough for you. If doesn't, I would recommend work with ModSecurity rule exceptions features first before decide to move to Collaborative Detection Mode.
It is important to mention that you should see results from performance point of view only if SecRuleEngine On.
4 – Rule Pre-Filtering
If you are planning to write your own rules, specially using@rx operator with nontrivial regex to inspect large amount of data like response bodies, you should consider use @pm operator as a pre-filter rule:
SecRule RESPONSE_BODY "@pm some_leak_patterns" "phase:4,chain,id:12345,deny"SecRule RESPONSE_BODY "@rxyour_nontrivial_regex_some_leak_patterns"
The @pm operator uses a fast multi-pattern match algorithm called Aho-Corasick and can be used to avoid execution of your regex against all inbound and outbound buffers.
Another pre-filter ideas are:
- Immediately reject IPs from countries
- Immediately reject IPs with bad reputation
- Immediately reject transactions with not allowed number of arguments
- Immediately reject transactions with not allowed arguments length.
With this idea in mind you can build a small set of rules that will run before the CRS and will immediately reject transactions to avoid them being inspected against all CRS rules.
5 – Buffering
ModSecurity works buffering inbound and outbound data to be later inspected by rules. The main bottleneck related to this topic is buffering response bodies for two reasons: it will consume a lot of RAM and usually rules placed in response body phase are expensive.
That being said you can consider disabling response body inspection setting SecResponseBodyAccess Off and conditionally enable it using ctl:responseBodyAccessOn during some specific situation for a certain types SecResponseBody Mime Types..
For example:
SecResponseBodyMimeType text/plain text/html text/xml
SecResponseBodyAccess Off
SecRule REQUEST_BODY|ARGS "@pm union select""phase:2,chain,id:1234,ctl:responseBodyAccess=On"
SecRule REQUEST_BODY|ARGS "@rxyour_nontrivial_regex_union_select"
SecRule RESPONSE_BODY "@pm some_leak_patterns""phase:4,chain,id:12345,deny"
SecRule RESPONSE_BODY "@rxyour_nontrivial_regex_some_leak_patterns"
6 – Logging
The Logging Engine could be a performance killer if you don't keep attention on:
Execute – constantly - a tuning process in your rules to avoid too much false positives. Keep in mind that disk i/o is expensive and you don't want to spend resource logging false positives.
Do not use serial logging mode. It uses locks to protect the file and will kill the performance. Use concurrent mode instead.
Review the audit log parts you are logging. Some parts like K and E can increase the overhead caused by the logging engine, because it usually needs to write big amount of data to disk.
Do not enable debug log in production.
7 – PCRE-JIT
A Just-In-Time compiler support was inserted into pc relibrary (>=8.20). Just-in-time compiling is a optimization that can greatly speed up pattern matching operations. It is of most benefit when the same pattern is going to be matched many times.
Release 8.20 21-Oct-2011 ------------------------
The main change in this release is the inclusion of ZoltanHerczeg's just-in-time compiler support, which can be accessed by building PCREwith --enable-jit. Large performance benefits can be had in many situations.8.20 also fixes an unfortunate bug that was introduced in 8.13 as well astidying up a number of infelicities and differences from Perl.
ModSecurity 2.7.x series can execute a regex using PCRE-JIT. This is a very good feature from performance point of view. To enable it you must compile ModSecurity using the follow configure option:
./configure –enable-pcre-jit
Make sure your PCRE library was compiled with JIT support(using the option described in the above pcre release notes). Also ModSecurity and Apache must use the same library version. You can check it looking into error log.
As an example we sent a large input to be processed by request body rules and measured the spent time:
JIT Disabled Phase 2 rules = 422749 usecs
JIT Enabled Phase 2 rules = 115777 usecs
Looking this example pcre-jit can make rules 75% faster on average.
8 – Caching Lua VM
This is for people that need to execute multiple Lua scripts in the same transaction. Normally ModSecurity will create and destroy a VM for each lua script running in the same transaction. You can change this behavior recompiling ModSecurity with the option:
./configure –enable-lua-cache
Once recompiled it, ModSecurity will keep in memory the Lua VM during the whole transaction, reducing the overhead caused by create/destroy VM operations.
As an example let's measure the performance of three scripts executing in the same transaction:
Cache disabled
Lua: Executing script: /etc/apache2/modsecurity/script1.lua
Lua: Script completed in 742 usec, returning: 1.
Lua: Executing script: /etc/apache2/modsecurity/script2.lua
Lua: Script completed in 517 usec, returning: 1.
Lua: Executing script: /etc/apache2/modsecurity/script3.lua
Lua: Script completed in 489 usec, returning: 1.
Total: 1748usecs
Cache enabled
Lua: Executing script: /etc/apache2/modsecurity/script1.lua
Lua: Script completed in 592 usec, returning: 1.
Lua: Executing script: /etc/apache2/modsecurity/script2.lua
Lua: Script completed in 130 usec, returning: 1.
Lua: Executing script: /etc/apache2/modsecurity/script3.lua
Lua: Script completed in 101 usec, returning: 1.
Total: 823usecs
We can see an overhead reduction of ~50% caused by VM create/destroy operations.
9 – Detecting expensive rules.
If there are more performance issues you can try the follow steps to find expensive rules and then have a chance to make it better.
As an example let's suppose we are trying to detect expensive rules running on phase 2 (request body). If you are using the latest ModSecurity version (>= 2.7.4) you have all those features to work.
Create a rule to detect if phase 2 is the problem. We will assume 1000 microseconds too much
SecRule PERF_PHASE2 "@qt 1000" "id:1234,phase:3"
If you have a trigger it is indicating you have expensive rules. So let's try to detect them adding the follow rules
# All rules that spent more than 50usecs will be present in audit log part H
SecRulePerfTime 50
# PERF_RULES is a collection that will contain all rules that spent more than SecRulePerfTime vallue to run.
SecRule PERF_RULES "@qt 50" "id:1,phase:3"
So if you have expensive rules we will see alerts:
[Tue Apr 23 17:50:26 2013] [error] [client 192.168.0.103]ModSecurity: Warning. Operator GT matched 50 at PERF_RULES:960032. [file"/etc/apache2/modsecurity/owasp-modsecurity-crs-2.2.6/base_rules/modsecurity_crs_99_perl.conf"][line "2"] [id "1"] [hostname "192.168.0.104"][uri "/acao.php"] [unique_id "UXcCIsCoAGUAACvOLqcAAAAD"]
[Tue Apr 23 17:50:26 2013] [error] [client 192.168.0.103]ModSecurity: Warning. Operator GT matched 50 at PERF_RULES:958022. [file"/etc/apache2/modsecurity/owasp-modsecurity-crs-2.2.6/base_rules/modsecurity_crs_99_perl.conf"][line "2"] [id "1"] [hostname "192.168.0.104"][uri "/acao.php"] [unique_id "UXcCIsCoAGUAACvOLqcAAAAD"]
[Tue Apr 23 17:50:26 2013] [error] [client 192.168.0.103]ModSecurity: Warning. Operator GT matched 50 at PERF_RULES:973323. [file"/etc/apache2/modsecurity/owasp-modsecurity-crs-2.2.6/base_rules/modsecurity_crs_99_perl.conf"][line "2"] [id "1"] [hostname "192.168.0.104"][uri "/acao.php"] [unique_id "UXcCIsCoAGUAACvOLqcAAAAD"]
As we can see rules 960032, 958022 and 973323 are spending more than 50usecs to execute., but how much time exactly ? You will get this information into audit log part H:
Rules-Performance-Info: "960032=622","958022=731", "973323=109".
Please check the Reference Manual to see all PERF_variables.
10 – Persistent Storage
ModSecurity's persistent storage mechanism is disk based. That being said it is not fast as if we could share data in memory. So were commend, if possible, setup and define your data directory in a RAMDISK. Persistent storage files in ModSecurity can be bigger than necessary because old entries will only overwritten when they are expired. By default the expiration time is 3600 seconds, It is usually too much, you probably want data for a few minutes, so you can reduce the default timeout using SecCollection Timeout directives.
