Trustwave achieves verified MXDR solution and FastTrack ready partner status from Microsoft. Learn More

Trustwave achieves verified MXDR solution and FastTrack ready partner status from Microsoft. Learn More

Managed Detection & Response

Eradicate cyberthreats with world-class intel and expertise

Managed Security Services

Expand your team’s capabilities and strengthen your security posture

Consulting & Professional Services

Tap into our global team of tenured cybersecurity specialists

Penetration Testing

Subscription- or project-based testing, delivered by global experts

Database Security

Get ahead of database risk, protect data and exceed compliance requirements

Email Security & Management

Catch email threats others miss with layered security & maximum control

Co-Managed SOC (SIEM)

Eliminate alert fatigue, focus your SecOps team, stop threats fast, and reduce cyber risk

Microsoft Exchange Server Attacks
Stay protected against emerging threats
Rapidly Secure New Environments
Security for rapid response situations
Securing the Cloud
Safely navigate and stay protected
Securing the IoT Landscape
Test, monitor and secure network objects
Why Trustwave
The Trustwave Approach
Awards and Accolades
Trustwave SpiderLabs Team
Trustwave Fusion Platform
SpiderLabs Fusion Center
Security Operations Centers
Technology Alliance Partners
Key alliances who align and support our ecosystem of security offerings
Trustwave PartnerOne Program
Join forces with Trustwave to protect against the most advance cybersecurity threats

Detecting Malicious Behavior by Unmasking WebSockets

WebSockets allow a single TCP connection to have full duplexing communications.  This type of connection reduces the overhead of HTTP polling, where the client would have to constantly request information from the server in order to get updates.  One of the features of WebSocket connection is the ability for the client to mask its payload when sending messages to the server.  Since this feature can also be used to hide malicious traffic, we’ll use Suricata’s Lua engine to unmask and inspect WebSocket payloads.

You’d think the masking was done to hide the clear text data in HTTP but it was actually put in place to protect against cache poisoning.  According to RFC 6455, an attacker could send what looked like a GET request to a server under their control and respond so that it looks like a typical response to that request.  There is the potential to poison the caches for a particular user and anyone else using the same one.

This masking can create a barrier to detecting malicious activity directed at servers using web sockets.  Luckily enough, all the information is present in the frame for us to decode the payload and take a peek inside.

Wireshark will automatically decode these payloads for you.  All you need to do is select the “Unmasked data” tab at the bottom.  The first two bytes in the payload set various flags and indicate the size.  The “Mask” flag is the one that needs to be checked in order for there to be masking used.  The 4 bytes right after the payload length are used to XOR the rest of the data.


Picture1Figure 1: masked Websockets data


Figure 2: unmasked Websockets data



Figure 3: Websockets decoded by Wireshark

The formula for both encoding and decoding the data is the same.  According to the RFC, the formula is as follows:

j = i MOD 4
transformed-octet-i = original-octet-i XOR masking-key-octet-j

If the above doesn’t quite make sense then here is some pseudo code that is similar to the Lua script we’ll see later. There will be some differences because of the way Lua handles array positions.

for i=0, i<len(payload), i++
key = arr_of_keys[i % 4]
decoded_data = payload[i] ^ key

CVE-2018-1270 is a good example of needing to inspect masked WebSocket traffic.  In this example, the exploit is using java.lang.Runtime to execute a Netcat command.  The images below show both the masked and unmasked traffic.  Since the traffic is masked when the IDPS sees it, a normal signature isn’t going to be useful without some kind of decoding first.


Picture4Figure 4: CVE-2018-1270 exploit masked by WebSockets


Figure 5: CVE-2018-1270 exploit unmasked


The signature below checks to see if this is a masked WebSocket request and, if so, calls the webscoket_decode.lua script.  You could add another signature to this and check for the `Upgrade: websocket` header in the server response and combine them with flowbits.  This could reduce the number of times the signature would call the Lua script and be more certain of the type of traffic being passed to the script.  We’ll keep it simple and use just one signature.

alert tcp any any -> any 8080 (msg:"Pivotal Spring Framework Remote Code Execution (CVE-2018-1270)"; 
flow:established,to_server; content:"|81|"; depth:1; byte_test:1,&,0x80,0; luajit:scripts/websocket_decode.lua;
reference:url, sid:20181270; rev:1;)

The init() function of the lua script grabs whichever buffer is needed.  In this case the payload is what we’ll need to unmask the data.  After that we move into the match() function and check to make sure there’s data in the buffer.  If there is then we can extract a 4 byte key using substring from the payload. Remember that all arrays in Lua begin with 1 not 0.

function init (args)
local needs = {}
needs["payload"] = tostring(true)
return needs

function match(args)
local a = tostring(args["payload"])
if #a > 0 then
local key = string.sub(a,3,6)
local decoded_payload = decode_payload(a, key)
if string.find(decoded_payload, 'java.lang.Runtime.*exec') then
return 1
return 0
return 0

The payload and key then gets passed to the decode_payload() function. In this function we start decoding right after the 4 byte key, so we set i in the for loop to 7.  In order to make sure we are using the correct byte in the key array this is typically done by key[i%4] which would give us a result between 0-3.  To make sure this aligns properly we just subtract 7 from i.  After that we XOR each byte with the appropriate key and store the results in an array.  Once the array is complete we need to convert it back to a string to make things easier to search with.

function decode_payload(a, key)
local tohex = bit.tohex
local unmask = {}
for i=7,#a,1
xor_key = string.byte(key, mod(i-7,4))
unmask[i-6] = string.char(bit.bxor(string.byte(a,i),xor_key))
local unmasked_string = arr_to_str(unmask)
return unmasked_string

Once we have a string we can search the data using any keyword we’d like.  For instance, I can use `string.find()` and use regex to look for `java.lang.Runtime` and the `exec` command to trigger the alert.

        if string.find(decoded_payload, 'java.lang.Runtime.*exec') then
return 1
return 0

While Suricata’s signatures have many options and can detect many things on their own, there are some circumstances where more advanced logic is needed.  Masked WebSockets are a great example of that and is one of many uses of Suricata’s Lua scripting engine.