Trustwave SpiderLabs Uncovers Ov3r_Stealer Malware Spread via Phishing and Facebook Advertising. Learn More

Trustwave SpiderLabs Uncovers Ov3r_Stealer Malware Spread via Phishing and Facebook Advertising. Learn More

Managed Detection & Response

Eliminate active threats with 24/7 threat detection, investigation, and response.

Co-Managed SOC (SIEM)

Maximize your SIEM investment, stop alert fatigue, and enhance your team with hybrid security operations support.

Advisory & Diagnostics

Advance your cybersecurity program and get expert guidance where you need it most.

Penetration Testing

Test your physical locations and IT infrastructure to shore up weaknesses before exploitation.

Database Security

Prevent unauthorized access and exceed compliance requirements.

Email Security

Stop email threats others miss and secure your organization against the #1 ransomware attack vector.

Digital Forensics & Incident Response

Prepare for the inevitable with 24/7 global breach response in-region and available on-site.

Firewall & Technology Management

Mitigate risk of a cyberattack with 24/7 incident and health monitoring and the latest threat intelligence.

Microsoft Exchange Server Attacks
Stay protected against emerging threats
Rapidly Secure New Environments
Security for rapid response situations
Securing the Cloud
Safely navigate and stay protected
Securing the IoT Landscape
Test, monitor and secure network objects
Why Trustwave
About Us
Awards and Accolades
Trustwave SpiderLabs Team
Trustwave Fusion Security Operations Platform
Trustwave Security Colony
Technology Alliance Partners
Key alliances who align and support our ecosystem of security offerings
Trustwave PartnerOne Program
Join forces with Trustwave to protect against the most advance cybersecurity threats
SpiderLabs Blog

Detecting Malicious Behavior by Unmasking WebSockets

WebSockets allow a single TCP connection to have full duplexing communications.  This type of connection reduces the overhead of HTTP polling, where the client would have to constantly request information from the server in order to get updates.  One of the features of WebSocket connection is the ability for the client to mask its payload when sending messages to the server.  Since this feature can also be used to hide malicious traffic, we’ll use Suricata’s Lua engine to unmask and inspect WebSocket payloads.

You’d think the masking was done to hide the clear text data in HTTP but it was actually put in place to protect against cache poisoning.  According to RFC 6455, an attacker could send what looked like a GET request to a server under their control and respond so that it looks like a typical response to that request.  There is the potential to poison the caches for a particular user and anyone else using the same one.

This masking can create a barrier to detecting malicious activity directed at servers using web sockets.  Luckily enough, all the information is present in the frame for us to decode the payload and take a peek inside.

Wireshark will automatically decode these payloads for you.  All you need to do is select the “Unmasked data” tab at the bottom.  The first two bytes in the payload set various flags and indicate the size.  The “Mask” flag is the one that needs to be checked in order for there to be masking used.  The 4 bytes right after the payload length are used to XOR the rest of the data.


Picture1Figure 1: masked Websockets data


Figure 2: unmasked Websockets data



Figure 3: Websockets decoded by Wireshark

The formula for both encoding and decoding the data is the same.  According to the RFC, the formula is as follows:

j = i MOD 4
transformed-octet-i = original-octet-i XOR masking-key-octet-j

If the above doesn’t quite make sense then here is some pseudo code that is similar to the Lua script we’ll see later. There will be some differences because of the way Lua handles array positions.

for i=0, i<len(payload), i++
key = arr_of_keys[i % 4]
decoded_data = payload[i] ^ key

CVE-2018-1270 is a good example of needing to inspect masked WebSocket traffic.  In this example, the exploit is using java.lang.Runtime to execute a Netcat command.  The images below show both the masked and unmasked traffic.  Since the traffic is masked when the IDPS sees it, a normal signature isn’t going to be useful without some kind of decoding first.


Picture4Figure 4: CVE-2018-1270 exploit masked by WebSockets


Figure 5: CVE-2018-1270 exploit unmasked


The signature below checks to see if this is a masked WebSocket request and, if so, calls the webscoket_decode.lua script.  You could add another signature to this and check for the `Upgrade: websocket` header in the server response and combine them with flowbits.  This could reduce the number of times the signature would call the Lua script and be more certain of the type of traffic being passed to the script.  We’ll keep it simple and use just one signature.

alert tcp any any -> any 8080 (msg:"Pivotal Spring Framework Remote Code Execution (CVE-2018-1270)"; 
flow:established,to_server; content:"|81|"; depth:1; byte_test:1,&,0x80,0; luajit:scripts/websocket_decode.lua;
reference:url, sid:20181270; rev:1;)

The init() function of the lua script grabs whichever buffer is needed.  In this case the payload is what we’ll need to unmask the data.  After that we move into the match() function and check to make sure there’s data in the buffer.  If there is then we can extract a 4 byte key using substring from the payload. Remember that all arrays in Lua begin with 1 not 0.

function init (args)
local needs = {}
needs["payload"] = tostring(true)
return needs

function match(args)
local a = tostring(args["payload"])
if #a > 0 then
local key = string.sub(a,3,6)
local decoded_payload = decode_payload(a, key)
if string.find(decoded_payload, 'java.lang.Runtime.*exec') then
return 1
return 0
return 0

The payload and key then gets passed to the decode_payload() function. In this function we start decoding right after the 4 byte key, so we set i in the for loop to 7.  In order to make sure we are using the correct byte in the key array this is typically done by key[i%4] which would give us a result between 0-3.  To make sure this aligns properly we just subtract 7 from i.  After that we XOR each byte with the appropriate key and store the results in an array.  Once the array is complete we need to convert it back to a string to make things easier to search with.

function decode_payload(a, key)
local tohex = bit.tohex
local unmask = {}
for i=7,#a,1
xor_key = string.byte(key, mod(i-7,4))
unmask[i-6] = string.char(bit.bxor(string.byte(a,i),xor_key))
local unmasked_string = arr_to_str(unmask)
return unmasked_string

Once we have a string we can search the data using any keyword we’d like.  For instance, I can use `string.find()` and use regex to look for `java.lang.Runtime` and the `exec` command to trigger the alert.

        if string.find(decoded_payload, 'java.lang.Runtime.*exec') then
return 1
return 0

While Suricata’s signatures have many options and can detect many things on their own, there are some circumstances where more advanced logic is needed.  Masked WebSockets are a great example of that and is one of many uses of Suricata’s Lua scripting engine.

Latest SpiderLabs Blogs

Trustwave SpiderLabs: Insights and Solutions to Defend Educational Institutions Against Cyber Threats

Security teams responsible for defending educational institutions at higher education and primary school levels often find themselves facing harsh lessons from threat actors who exploit the numerous...

Read More

Breakdown of Tycoon Phishing-as-a-Service System

Just weeks after Trustwave SpiderLabs reported on the Greatness phishing-as-a-service (PaaS) framework, SpiderLabs’ Email Security team is tracking another PaaS called Tycoon Group.

Read More

Trustwave SpiderLabs Uncovers Ov3r_Stealer Malware Spread via Phishing and Facebook Advertising

During an Advanced Continual Threat Hunt (ACTH) investigation that took place in early December 2023, Trustwave SpiderLabs discovered Ov3r_Stealer, an infostealer distributed using Facebook...

Read More