Trustwave SpiderLabs Uncovers Ov3r_Stealer Malware Spread via Phishing and Facebook Advertising. Learn More

Trustwave SpiderLabs Uncovers Ov3r_Stealer Malware Spread via Phishing and Facebook Advertising. Learn More

Managed Detection & Response

Eliminate active threats with 24/7 threat detection, investigation, and response.

Co-Managed SOC (SIEM)

Maximize your SIEM investment, stop alert fatigue, and enhance your team with hybrid security operations support.

Advisory & Diagnostics

Advance your cybersecurity program and get expert guidance where you need it most.

Penetration Testing

Test your physical locations and IT infrastructure to shore up weaknesses before exploitation.

Database Security

Prevent unauthorized access and exceed compliance requirements.

Email Security

Stop email threats others miss and secure your organization against the #1 ransomware attack vector.

Digital Forensics & Incident Response

Prepare for the inevitable with 24/7 global breach response in-region and available on-site.

Firewall & Technology Management

Mitigate risk of a cyberattack with 24/7 incident and health monitoring and the latest threat intelligence.

Microsoft Exchange Server Attacks
Stay protected against emerging threats
Rapidly Secure New Environments
Security for rapid response situations
Securing the Cloud
Safely navigate and stay protected
Securing the IoT Landscape
Test, monitor and secure network objects
Why Trustwave
About Us
Awards and Accolades
Trustwave SpiderLabs Team
Trustwave Fusion Security Operations Platform
Trustwave Security Colony
Technology Alliance Partners
Key alliances who align and support our ecosystem of security offerings
Trustwave PartnerOne Program
Join forces with Trustwave to protect against the most advance cybersecurity threats
SpiderLabs Blog

WAF Normalization and I18N

Submitted By Breno Silva Pinto and Ryan Barnett

WAF Normalization and I18N

Web application firewalls must be able to handle Internationaliztion (I18N) and thus properly handle various data encodings including Unicode and UTF-8 in order to prevent not only evasion issues but also to minimize false positives. In an earlier blog post, we highlighted ModSecurity's new support for Unicode mapping and decoding. This capability helps us to more accurately decode characters from different Unicode code points. While this certainly helps our accuracy, we still had the issue of UTF-8 encodings.. This is a challenge for any WAF as it must be able to handle UTF-8 encodings of characters for different languages such as Portuguese. So, if you are running ModSecurity to protect a non-English language website then this blog post is for you! We introduce a new transformation function called utf8toUnicode that helps to normalize data for inspection.

Incorrect UTF-8 Decoding

We have received some recent reports of false positive issues with the OWASP ModSecurity CRS that were due to existence of UTF-8 encoded characters. As an example, the Portuguese language has many "special" characters with different accent characters. For example:

1-    á Á  ã Ã â Â à À
é É ê Ê í Í
ó Ó õ Õ ô Ô
ú Ú ü Ü
ç Ç

When these characters are UTF-8 encoded, they use multiple bytes. As an example, the "ę" character is encoded as "%c4%99". If ModSecurity only applies the standard t:urlDecodeUni, it will decode each byte individually which results in an impedance mismatch. In this case, this incorrect decoding resulted in a false positive match against some SQL Injection rules in the OWASP ModSecurity CRS. While this is a bit of a pain, it is not as bad as a false negative bypass situation that may be caused by this type of incorrect decoding . Let's look an this type of SQL Injection evasion issue. What if we send the following request:'úníón+séléct+data+fróm+námés

Let's see how ModSecurity will decode this data when checking for an example SQL Injection keyword:

Recipe: Invoking rule 21b67e38;
Rule 21b67e38: SecRule "ARGS:foo""@rx select""phase:2,log,auditlog,pass,id:1111,t:urlDecodeUni"
T (0) urlDecodeUni:"'\xc3\xban\xc3\xad\xc3\xb3n s\xc3\xa9l\xc3\xa9ctdata fr\xc3\xb3m n\xc3\xa1m\xc3\xa9s"
Transformation completed in 13 usec.
Executing operator "rx" withparam "select" against ARGS:foo.
Target value:"'\xc3\xban\xc3\xad\xc3\xb3n s\xc3\xa9l\xc3\xa9ct data fr\xc3\xb3mn\xc3\xa1m\xc3\xa9s"
Operator completed in 7 usec.
Rulereturned 0

As you can see above the character sequence "úníón+séléct+data+fróm+námés" was handled by the engine as "\xc3\xban\xc3\xad\xc3\xb3ns\xc3\xa9l\xc3\xa9ct data fr\xc3\xb3m n\xc3\xa1m\xc3\xa9s". However the rulewas looking for the pattern "select". So, what happen if the application applies best-fit mapping conversions and removes the accents before sending the data tothe database ? This may allow the payload to bypass our signatures.

Utf8 to Unicode Mapping

In order to better handle this data, we should first map the UTF-8 encoded data to Unicode and then use the unicode point mapping capabilities mentioned at the beginning of the blog post. This configuration is achieved by first setting the SecUnicodeCodePage and SecUnicodeMapFile directives in your main ModSecurity configuration file:

SecUnicodeCodePage 20127SecUnicodeMapFile /etc/apache2/unicode.mapping

The SecUnicodeCodePage directive sets the proper Unicode code point used for your site. The example 20127 is the US-ASCII code point. The SecUnicodeMapFile points to the unicode.mapping file that comes with the ModSecurity source archive and includes all of the Unicode conversions. Here is an example of the 20127 (US-ASCII) mapping data:

20127 (US-ASCII)00a0:20 00a1:21 00a2:63 00a4:24 00a5:59 00a6:7c 00a9:43 00aa:6100ab:3c 00ad:2d 00ae:52 00b2:32 00b3:33 00b7:2e 00b8:2c 00b9:31 00ba:6f 00bb:3e00c0:41 00c1:41 00c2:41 00c3:41 00c4:41 00c5:41 00c6:41 00c7:43 00c8:45 00c9:4500ca:45 00cb:45 00cc:49 00cd:49 00ce:49 00cf:49 00d0:44 00d1:4e 00d2:4f 00d3:4f00d4:4f 00d5:4f 00d6:4f 00d8:4f 00d9:55 00da:55 00db:55 00dc:55 00dd:59 00e0:6100e1:61 00e2:61 00e3:61 00e4:6100e5:61 00e6:61 00e7:63 00e8:6500e9:65 00ea:65 00eb:65 00ec:69 00ed:69 00ee:69 00ef:69 00f1:6e 00f2:6f 00f3:6f00f4:6f 00f5:6f 00f6:6f 00f8:6f 00f9:75 00fa:75 00fb:75 00fc:75 00fd:79 00ff:790100:41 0101:61 0102:41 0103:61 0104:41 0105:61 0106:43 0107:63 0108:43 0109:63010a:43 010b:63 010c:43 010d:63 010e:44 010f:64 0110:44 0111:64 0112:45 0113:650114:45 0115:65 0116:45 0117:65 0118:45 0119:65 011a:45 011b:65 011c:47 011d:67011e:47 011f:67 0120:47 0121:67 0122:47 0123:67 0124:48 0125:68 0126:48 0127:680128:49 0129:69 012a:49 012b:69 012c:49 012d:69 012e:49 012f:69 0130:49 0131:690134:4a 0135:6a 0136:4b 0137:6b 0139:4c 013a:6c 013b:4c 013c:6c 013d:4c 013e:6c0141:4c 0142:6c 0143:4e 0144:6e 0145:4e 0146:6e 0147:4e 0148:6e 014c:4f 014d:6f014e:4f 014f:6f 0150:4f 0151:6f 0152:4f 0153:6f 0154:52 0155:72 0156:52 0157:720158:52 0159:72 015a:53 015b:73 015c:53 015d:73 015e:53 015f:73 0160:53 0161:730162:54 0163:74 0164:54 0165:74 0166:54 0167:74 0168:55 0169:75 016a:55 016b:75016c:55 016d:75 016e:55 016f:75 0170:55 0171:75 0172:55 0173:75 0174:57 0175:770176:59 0177:79 0178:59 0179:5a 017b:5a 017c:7a 017d:5a 017e:7a 0180:62 0189:440191:46 0192:66 0197:49 019a:6c 019f:4f 01a0:4f 01a1:6f 01ab:74 01ae:54 01af:5501b0:75 01b6:7a 01cd:41 01ce:61 01cf:49 01d0:69 01d1:4f 01d2:6f 01d3:55 01d4:7501d5:55 01d6:75 01d7:55 01d8:75 01d9:55 01da:75 01db:55 01dc:75 01de:41 01df:6101e4:47 01e5:67 01e6:47 01e7:67 01e8:4b 01e9:6b 01ea:4f 01eb:6f 01ec:4f 01ed:6f01f0:6a 0261:67 02b9:27 02ba:22 02bc:27 02c4:5e 02c6:5e 02c8:27 02cb:60 02cd:5f02dc:7e 0300:60 0302:5e 0303:7e 030e:22 0331:5f 0332:5f 2000:20 2001:20 2002:202003:20 2004:20 2005:20 2006:20 2010:2d 2011:2d 2013:2d 2014:2d 2018:27 2019:27201a:2c 201c:22 201d:22 201e:22 2022:2e 2026:2e 2032:27 2035:60 2039:3c 203a:3e2122:54 ff01:21 ff02:22 ff03:23 ff04:24 ff05:25 ff06:26 ff07:27 ff08:28 ff09:29ff0a:2a ff0b:2b ff0c:2c ff0d:2d ff0e:2e ff0f:2f ff10:30 ff11:31 ff12:32 ff13:33ff14:34 ff15:35 ff16:36 ff17:37 ff18:38 ff19:39 ff1a:3a ff1b:3b ff1c:3c ff1d:3dff1e:3e ff20:40 ff21:41 ff22:42 ff23:43 ff24:44 ff25:45 ff26:46 ff27:47 ff28:48ff29:49 ff2a:4a ff2b:4b ff2c:4c ff2d:4d ff2e:4e ff2f:4f ff30:50 ff31:51 ff32:52ff33:53 ff34:54 ff35:55 ff36:56 ff37:57 ff38:58 ff39:59 ff3a:5a ff3b:5b ff3c:5cff3d:5d ff3e:5e ff3f:5f ff40:60 ff41:61 ff42:62 ff43:63 ff44:64 ff45:65 ff46:66ff47:67 ff48:68 ff49:69 ff4a:6a ff4b:6b ff4c:6c ff4d:6d ff4e:6e ff4f:6f ff50:70ff51:71 ff52:72 ff53:73 ff54:74 ff55:75 ff56:76 ff57:77 ff58:78 ff59:79 ff5a:7aff5b:7b ff5c:7c ff5d:7d ff5e:7e

With these Unicode mapping directives in place, we now can use the new t:utf8toUnicode transformation function which was just added to the v2.7.0 code. Here is a new example rule:

SecRule "ARGS:foo" "@rx select" "phase:2,log,auditlog,pass,id:1111,t:utf8toUnicode,t:urlDecodeUni"

Let's see how ModSecurity will handle the same request, but nowusing the mentioned features:

Recipe: Invoking rule 21717d58;
Rule 21717d58: SecRule "ARGS:foo""@rx select""phase:2,log,auditlog,pass,id:1111,t:utf8toUnicode,t:urlDecodeUni"
T (0) Utf8toUnicode:"'%u00fan%u00ed%u00f3ns%u00e9l%u00e9ct data fr%u00f3m n%u00e1m%u00e9s"
T (0) urlDecodeUni:"'union select data from names"
Transformation completed in 29 usec.
Executing operator "rx" withparam "select" against ARGS:foo.
Target value: "'union select data fromnames"
Operator completed in 16 usec.
Warning. Pattern match "select"at ARGS:foo.
Rulereturned 1.

As you can see, the sequence "úníón+séléct+data+fróm+names" is now normalized to "union select data from names" and thus the rule matched. Summarizing the engine steps for data normalization:

  1. Attacker input: úníón+séléct+data+fróm+names.
  2. Input in UTF-8 format: \xc3\xban\xc3\xad\xc3\xb3n s\xc3\xa9l\xc3\xa9ct datafr\xc3\xb3m n\xc3\xa1m\xc3\xa9s
  3. Input in Unicode format: %u00fan%u00ed%u00f3n s%u00e9l%u00e9ct data fr%u00f3mn%u00e1m%u00e9s
  4. Input in ASCII format: union select data from names

With this new UTF-8 and Unicode mapping and decoding support, ModSecurity can now more accurately normalize data from non-english languages which results in better rule accuracy.

Latest SpiderLabs Blogs

Welcome to Adventures in Cybersecurity: The Defender Series

I’m happy to say I’m done chasing Microsoft certifications (AZ104/AZ500/SC100), and as a result, I’ve had the time to put some effort into a blog series that hopefully will entertain and inform you...

Read More

Trustwave SpiderLabs: Insights and Solutions to Defend Educational Institutions Against Cyber Threats

Security teams responsible for defending educational institutions at higher education and primary school levels often find themselves facing harsh lessons from threat actors who exploit the numerous...

Read More

Breakdown of Tycoon Phishing-as-a-Service System

Just weeks after Trustwave SpiderLabs reported on the Greatness phishing-as-a-service (PaaS) framework, SpiderLabs’ Email Security team is tracking another PaaS called Tycoon Group.

Read More