ModSecurity is an open-source WAF engine maintained by Trustwave. As a lively open-source project, we constantly work together with the community on reported bugs, feature requests, and other issues on the ModSecurity GitHub.
For those not familiar with open source projects or GitHub, the idea is that anyone can participate and contribute to making the project better. Most often people participate through reporting issues that they’ve encountered in order to help identify and resolve them to make ModSecurity better, and sometimes they also submit suggestions for code fixes to help solve these issues (“pull requests”).
One such pull request was submitted by Ervin Hegedüs (airween) to address a couple of cookie parsing issues and a couple of days later Andrea Menin (theMiddleBlue) added via private channels that one of these issues could lead to a Denial of Service (DoS) attack on a server running ModSecurity versions 3.0 up to 3.0.3.
Once our researchers confirmed the potential of a DoS we worked together with Ervin and Andrea to refine and ultimately provide a fix to address the parsing issues, as well as the DoS.
Below is information on where you can find the fix, as well as some more technical details about the vulnerability.
Exploits targeting this vulnerability would send a malformed HTTP “Cookie” header to a vulnerable server. This will cause an out_of_range exception which, in the common use case of ModSecurity used in conjunction with nginx, will crash the nginx worker thread (the thread responsible for handling the request). Continuously sending such requests to the server will repeatedly crash worker threads and in cases where requests are sent faster than worker threads can recover, it will cause a Denial of Service on the server.
Quite often when our researchers discover and disclose vulnerabilities in other products, they like to share not only the technical details of the vulnerability but also the story of how it was discovered. This is not just for the sake of good storytelling, these are lessons and points for others to pay attention to and think about the next time they find themselves in a similar situation.
In the case of this vulnerability, it started with Ervin working on a bug he discovered while running regression tests for the OWASP Core Rule Set (CRS), comparing ModSecurity v2 and v3 to find discrepancies in behaviours:
Ervin: “CRS has a regression test case, where the request contains only a single string without `=` as HTTP Cookie. The failed result with modsecurity3 occurred because if the Cookie header parts didn't contained `=`, then it wasn't handled as cookie.”
While investigating the code, Ervin noticed a couple of other issues and decided to work on a fix.
Ervin: “When I was done with the code and regression tests, I wanted to be sure that I thought all possible cookie strings. For the easiest check, I made a small utility, which contains the old and new cookie parsing method, and the related helper functions from the source of libmodsecurity3. This helped me to see the differences between the old and new implementation.”
Andrea: “Ervin reproduced both libModSecurity cookie parser and his patched version in a standalone binary that reads a cookie string from the first argument and parse it comparing the results between the "old" and the new parser. Ervin asked me to check his patch looking for other ways to bypass it. As you know, in a cookie string each name and value are separated by a = and each cookie name/value pairs are separated by a ;. After few tests, I started to put random sequences of ; and = trying to crash it. By sending ;=; nothing has happened but removing the first ; something went wrong.”
What they encountered was the aforementioned out_of_bounds exception.
Andrea: “Immediately we thought about how this ‘out of range’ could affect a running webserver and if it could lead to a Denial of Service. So, I've tested it on a docker container running Nginx + libModSecurity sending a request via curl [… ] but I didn't get any response from the web server, even the response headers part. By reading the Nginx error_log it becomes clear that my previous request killed the Nginx worker process that wrote the same "out of range" error on the log. From here, it was really easy for me to stuck Nginx just by sending multiple requests with curl and continuously killing all respawning Nginx worker processes.
Bugs are an inevitable part of code, and while no developer is ever truly happy to have an issue reported to them, knowing that an issue is fixed and one more piece of code is that much better, is ultimately a long-term positive for any project. For that, we’d like to thank Ervin and Andrea for reporting the vulnerability and working with us to disclose it in a responsible manner.
The great thing about open source projects is that the code is out there for all to see, review, and help improve. The more people who participate in the project the better it becomes.
We’d like to take this opportunity to encourage everyone who uses ModSecurity to participate. You don’t have to be a developer. If you’re a ModSecurity user and you open a GitHub issue with helpful details when you encounter a bug- you’re helping! If you test/confirm that a fix we implemented is doing what it’s supposed to- you’re helping! If you’re deploying ModSecurity in weird ways and it works and you tell us about it so we know someone tried it- you’re helping!
Thanks to those who already contribute and participate in these various ways and others, ModSecurity is better for it, and we’ll continue to strive and make it better with every release.