Help, my ModSecurity Community Console is not responding!"
Perhaps you have seen this type of email sent to the ModSecurity mailing list? Unfortunately, it is relatively easy to overload the ModSecurity Conmmunity Console when you combine the following three factors: using the default Console configurations, having the ModSecurity Sensor configured to log alerts for all traffic and if your site has large amounts of traffic. In this type of situation, the Console MUI may become unresponsive. This is due to having too many open/active alert messages. So, what can you do about it? Read on to find steps to help alleviate this problem.
SecAuditEngineThe SecAuditEngine Directive can have an impact on the total number of alert messages being created, mainly if it is set to On. The recommended setting for this directive is RelevantOnly.
SecAuditLogRelevantStatusThe defualt setting for this directive in the Core Rules is "^[45]". This directive will generate audit logs for all 4xx and 5xx level status codes. The downside of this setting, from a performance perspective, is that this also logs 404 Not Found status codes which can generate a lot of information. If you want to exclude 404 status codes from being logged, use this setting - "^(?:5|4\d[^4])".
modsecurity-auditlog-collectorThe Perl script that is currently available for transporting the concurrent audit log files to the remote Console host does not perform well under production loads. There is even a disclaimer saying as much inside the script:
# This code is not suitable for non-trivial production use
# since it can only submit one audit log entry at a time, plus
# it does not handle errors gracefully.
So, what can you do? Breach Security is putting the finishing touches on a production quality audit log transport replacement tool called mlogc. This tool will be released to the public in the very near future so keep your eyes out for it. It won't solve all of your performance issues but its will help.
The main memory hog in the Console is the number of active alerts (alerts with status open) in the Alert interface. Part of this can be alleviated by properly configuring the various ModSecurity directives mentioned previously as to not generate too many needless alert messages.
Another change that can be made to help avoid a memory problem associated with this issue is to update the Alert Management configuration settings. You can optionally go to Administration -> Alert Management in the MUI and decrease the "Automatic Stale Alert Removal" time setting. This will automatically archive and remove alerts below a specified alert severity from the alert page. This can help to guard against low level Notice/Info alerts from clogging up the Alert interface. You can still get to all of the data, however you would need to use the Transactions Search page to bring them back up.
If your Console is being bombarded by alert messages, the initial homepage may not be responsive when you login as the MUI is attempting to display the proper number of Active Alerts under the Sensor Overview section. If this is happening and your MUI is hanging, try accessing another page that does not display the Active Alert data such as the Administration page located here - http://CONSOLEHOST:8886/adminHome. This should allow you to get access to the Alert Management interface (mentioned in the previous section) and possibly flush out the active alerts and archive them to the database.
You can allocate more memory available/allocated to the Java MUI application. The amount of Memory (RAM) that is allocated to the Java application can be increased by editing the modsecurity-console shell script file. You want to edit the -Xms and -Xmx arguments in the following line:
-INSTALL4J_ADD_VM_PARAMS="-Xms128M -Xmx512M - Dderby.storage.pageSize=32768 -Dderby.storage.pageCacheSize=8000"
Here is a quick breakdown of what these parameters mean:
-Xms128M - start with 128 MB RAM-Xmx512M - use max. 512 MB
If you increase these values, you should take care to still leave some RAM to be used by the operating system and other processes on the box.
-Dderby.storage.pageCacheSize=8000 - each page should be 8 KB-Dderby.storage.pageSize=32768 - have 32768 pages in cache
In the above example you allocate 32768 x 8 KB = 256 MB. To be on the conservative side, the database cache should be at most 30% of the total RAM dedicated to the Java process. Feel free to experiment to find the optimal value for your case.
If your site simply generates too much traffic for the free Console to handle, you many want to consider purchasing the soon to be released ModSecurity Management Console provided by Breach Security. This updated version of the Console has much better performance. It will initially function just like the open source Console in that it is used as a centralized alert management host. In future releases, it will also include "Command and Control" functionality that will allow it to actually manage remote ModSecurity Sensors.