Blogs & Stories

SpiderLabs Blog

Attracting more than a half-million annual readers, this is the security community's go-to destination for technical breakdowns of the latest threats, critical vulnerability disclosures and cutting-edge research.

(Updated) Advanced Topic of the Week: Handling Authorized Scanning Traffic

Updated - the latest OWASP ModSecurity CRS release has a rules file to handle Authorized Vulnerability Scanning traffic.

Once you have deployed ModSecurity, you have probably been faced with this question:

How should I configure my Web Application Firewall (WAF) to handle Authorized Vulnerability Scanning (AVS) traffic?

The answer to this question is not quite as easy as it may first appear. This question arises when organizations are running their own internal web application vulnerability scans. They soon realize that they need to figure out how to get their security tools (scanner and waf) to "play nice" with each other.

Before deciding on how to reconfigure ModSecurity with regards to handling the scanning traffic, you first must confirm the goal of your scanning efforts. There are usually two main scanning goals:

  • To identify all vulnerabilities within a target web application, or

  • To identify all vulnerabilities within a target web application that are remotely exploitable by an external attacker.

You may want to reread the seconed item to make sure that you understand the difference, as it is factoring in the exploitability of a vulnerability in a production web application.

Identifying Underlying Vulnerabilities

Identifying the underlying vulnerabilities is a critical goal to have when scanning. If you can identify all of the existing vulnerabilities, or at least those which can be identified by scanning, you can then formulate a remediation plan to address the issue at the root cause. Remediation should include both source code fixes and custom rules within a web application firewall such as ModSecurity for protection in the interim.

In order to identify all of the actual vulnerabilities, you will need to allow the vulnerability scanning host to interact completely with the back-end web application. The advantages of this approach are twofold:

  • First, you will have the best chance to identify all of vulnerabilities and information leakages within the web application, and

  • Second, from a ModSecurity event management perspective, this configuration will eliminate alerts generated by the scans.

The downside of this approach is that it reduces the true exploitability accuracy of the scans as it does not simulate what a "normal" attacker would be able to access since the ModSecurity is not blocking inbound attacks and outbound information leakages. For whitelisting examples, please refer to the Implementing Exceptions - Whitelisting based on Client Location recipe.

ModSecurity Event Management

From a ModSecurity administration perspective, it is important to have proper configuration so as to allow for the vulnerability scanning goal while simultaneously not flooding the alert monitoring processes with thousands of events from authorized vulnerability scanning traffic. This data may impact both the overall performance of ModSecurity and it may skew reporting outputs that include raw attack data.

If you do not implement any specify policy changes to handle authorized scanning traffic, and ModSecurity is running with SecRuleEngine On to allow for disruptive blocking actions, then it will give a truer picture of what an attacker would or wouldn't be able to exploit since ModSecurity is providing its normal protections. In this configuration, however, the scans will possibly generate huge volumes of alerts especially if the scan frequency is high (daily). Additionally, vulnerabilities within the web application may not be identified and therefore are not being remediated through other means. It is for these reasons that you should consider the following scanning configuration policies.

Run Before and After Scans

Another scanning method to use is to coordinate with the vulnerability scanning team to conduct two different scans – one when the scanner is whitelisted and one when it is not. After the scans have been completed, the reduction of the identified vulnerabilities can almost assuredly be attributed to the ModSecurity blocking rule sets. The benefits of this approach are that it allows the scanning team to identify actually vulnerabilities, as well as, provides a tangible return on investment (ROI) of ModSecurity's protections. The drawback of this approach are that you have to run two scans in order to meet your desired scanning goals.

Block But Don't Log Configuration

One option you might want to consider and is a "block but don't log" configuration which is a hybrid approach that is suitable for most day to day usage. With this setup, you allow ModSecurity to block requests/responses when it sees attacks however in the phase:5 logging phase it will then inspect the IP address range and if it is coming from an authorized scanning host it will not generate alerts.

SecRule REMOTE_ADDR "@streq" \

While this rule is functionally correct, it could be improved from a performance perspective and moved from phase:5 to phase:1. This is a functionally equivalent rule, however the overall performance is improved since the audit engine is not burdened with creating internal audit events that will later be discarded.

# Recommended "Block but Don't Log" rule for scanning traffic
SecRule REMOTE_ADDR "@streq" \

Only Log Outbound Issues

Ideally, there should be a balance between both the vulnerability scanner and ModSecurity's viewpoints. This configuration certainly removed an audit log flooding scenario based on identified inbound attack traffic, however it is at the expense of ModSecurity's visibility of outbound monitoring. We don't really care about inbound attack attempts, however successful attack transactions are another story. If the vulnerability scanner was able to either send an attack payload that evaded the ModSecurity rulesets or if the protected web application generated some sort of error or information leakage, then there would be no audit log of the transaction. It is for this scenario that this final rule set is recommended:

# Recommended "Block but Don't Log" rule for scanning traffic
SecRule REMOTE_ADDR "@streq" \

# Recommended phase 3 rule that will re-enable the audit engine if the request
# was not blocked by one of the normal rules.
SecRule REMOTE_ADDR "@streq" \

This rule set toggles off the the audit engine in phase:1 if it is coming from the authorized scanning host IP address. If ModSecurity blocks an inbound attack, it will immediately proceed to phase:5 and then exit. If, on the otherhand, the scanner traffic does not trigger any inbound rules, then when it reaches phase:3 response headers, this rule will toggle the audit engine back on. This will allow ModSecurity to generate proper alerts for successful attacks and information leakage issues. In this final configuration, any audit events that are created based on the vulnerability scanning traffic should most certainly be investigated for remediation.