Debugging a program that communicates with a remote endpoint usually involves analyzing the network communications. A common method is capturing the traffic using a packet analyzer tool such as tcpdump or Wireshark. However, this process can be tricky when the communication is encrypted. Our team, responsible for the Trustwave network vulnerability scanning system, regularly encounters this situation - especially when scanning systems we don't control (mostly those of customers).
In this blog post, I will explain how to decrypt SSL/TLS communications to allow for the analysis of that traffic with Wireshark. I will focus on Ruby and the binding for OpenSSL. Please note that the topic of this post is not methods for breaking crypto-systems. Instead, it's about how to retrieve key material for decryption.
Decryption using Wireshark
Wireshark is one of the more famous packet analyzers out there, and it's capable of decrypting SSL/TLS communications. All it needs is the Master Key used in encrypting the data.
Here is a high level explanation of how Wireshark can retrieve this key:
- First, Wireshark checks whether any cached session matches the Session ID or the Session Ticket from the Server Hello handshake message. If so, it retrieves the Master Key from the cached session.
- If the key exchange algorithm is PSK, you can setup the path to the clear text Pre-Shared Key that was used during the key exchange. Wireshark will use it to calculate the Master Key.
- If the key exchange algorithm is RSA, you can provide the server Private Key (in PEM format) that was used for encryption during the key exchange step. Wireshark will try to decrypt the encrypted Pre-Master Key, which is retrieved from the Client Key Exchange handshake message. If successful, it uses the decrypted Pre-Master Key to calculate the Master Key.
- Finally, you can provide a key log file that contains a list of Master Keys and/or Pre-Master Keys (used to calculate the Master Key - the same way explained above).
Only step four is relevant in our case. Step one is mostly an internal Wireshark process and only possible if a previous session with the correct key material has been cached. This isn't relevant to our objective. We want to decrypt new SSL sessions. Steps two and three are limited to PSK and RSA key-exchange algorithms, which also require keys from the remote server (hard to capture in the context of vulnerability scanning).
The key log file
Without going deeper in the cryptographic process, Wireshark can decrypt the SSL communication using the Master Key exchanged during the handshake. Since a network capture (pcap) will likely contain many SSL sessions, it needs to be able to map the key with the corresponding SSL traffic. To do so, it provides a unique identifier for each key you put in the key log file. Each time a new SSL session begins, Wireshark will review the key log file entries to look for the identifier corresponding to the current session. If the identifier is found, it retrieves the corresponding key and decrypts the whole session.
The general format is "<id> <key>". Here is the definition from Wireshark source code documentation:
- "RSA xxxx yyyy": Where xxxx are the first eight bytes of the encrypted Pre-Master Secret (hex-encoded) and yyyy is the cleartext Pre-Master Secret (hex-encoded).
- "RSA Session-ID:xxxx Master-Key:yyyy": Where xxxx is the SSL session ID (hex-encoded) and yyyy is the cleartext Master Secret (hex-encoded).
- "CLIENT_RANDOM xxxx yyyy": Where xxxx is the Client Random from the ClientHello (hex-encoded) and yyyy is the cleartext Master Secret (hex-encoded).
For the first item, you need to provide the first eight bytes of the encrypted Pre-Master Secret, which will serve as an identifier.
In that case, Wireshark will do the following: extract the encrypted Pre-Master Secret from the Client Key Exchange handshake message in the pcap, look for the first eight bytes in the key log file, retrieve the plain text Pre-Master Secret and finally calculate the Master Key. The only issue with this method is it doesn't work with the Diffie-Hellman (DH) key exchange algorithm because the Pre-Master Secret is not kept anywhere.
The last two options work similarly: a unique identifier followed by the Master Key. One uses the Session ID ("RSA Session-ID: …" format) and the other the Client Random value ("CLIENT_RANDOM …" format). According to RFC 5246, the Session ID is "a value generated by a server that identifies a particular session" and the Client Random is "a 32-byte value provided by the client". The Session ID is used to resume a cached session and the Client Random is part of the key material needed to compute the Master Key (along with the Server Random and the Pre-Master Secret). Since both are generated for each SSL session, they can be used as an identifier.
Setting up Wireshark
You will find the SSL-related configuration in Edit menu under Preferences/Protocols (on the left hand side)/SSL (after expanding the tree menu).
Here are the interesting fields:
-
RSA keys list: this is where you can provide server private keys to decrypt the Pre-Master Secrets, but only if it was encrypted with the server public key (i.e., via the RSA key exchange algorithm).
-
SSL debug file: you can provide the path of a text file that will be used by Wireshark to output useful debugging information when decrypting SSL (recommended for troubleshooting).
-
Pre-Shared-Key: here is where the plain text Pre-Shared Key goes... duh! (but only when the PSK key exchange algorithm is used).
-
(Pre)-Master-Secret log filename: this is where the key log file path goes which we discussed in the previous section. We will go into more detail later.
Export Session ID in Ruby file
On the client side, a basic Ruby SSL connection is usually initiated as follows:
require 'socket'
require 'openssl'
socket = TCPSocket.open("192.168.1.214", 443)
ssl_context = OpenSSL::SSL::SSLContext.new
ssl_socket = OpenSSL::SSL::SSLSocket.new(socket, ssl_context)
ssl_socket.connect
Here we are using the default SSL context, and we connect to the port 443/TCP on the remote host (192.168.1.241).
Ruby offers SSL session information in three formats: DER, PEM and plaintext. The output of the #to_text method gives you the same output that "openssl s_client" will display after a successful connection:
> puts ssl_socket.session.to_text
SSL-Session:
Protocol : TLSv1
Cipher : AES256-SHA
Session-ID: EF306F66879F1FDE10EEA29E3D30BED1AED6A6894B7439A7F21829DE9DD2D9CB
Session-ID-ctx:
Master-Key: E6822B0C718654FDA14F2635689DD586D52AC686A0FFA31F6D3B309DEE5B9B8708107ED450C25E963DDCFE57F64CEF2E
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 92 79 20 40 4b ce 7b 9b-09 8b 82 40 8b 28 1e 7c .y @K.{....@.(.|
0010 - c5 08 f6 70 ac 99 30 c7-36 bc 48 dd 20 23 b0 b9 ...p..0.6.H. #..
0020 - 4d 40 40 a6 ed a6 e1 6a-75 2b b4 2a fb c5 72 01 M@@....ju+.*..r.
0030 - 15 fe 05 fa e2 70 db 37-c9 c6 ac 30 63 c0 6e b2 .....p.7...0c.n.
0040 - a4 84 2e 22 e0 e3 67 3f-9e 00 53 0f 21 76 d2 f5 ..."..g?..S.!v..
0050 - eb 72 c0 78 37 8a 4e 8b-1b 95 ee 12 80 0d 93 fb .r.x7.N.........
0060 - 54 73 90 bc bc 94 2d 23-7a 0b b2 e2 ba 3d 61 ec Ts....-#z....=a.
0070 - 45 06 4f bc c2 00 b0 67-ea 71 79 65 a3 5f 53 98 E.O....g.qye._S.
0080 - 05 ce 5f 96 5c c4 f7 55-30 22 b9 59 57 0f 01 cc .._.\..U0".YW...
0090 - 77 ed 54 8d eb 74 27 a8-3c bd 69 e0 bb c6 ad a7 w.T..t'.
As can be seen, the Session ID and the Master Key are displayed and already in the correct format. This is all we need to fill our key log file that Wireshark will use to decrypt this communication.
Here is a simple proof of concept:
socket = TCPSocket.open("192.168.1.214", 443)
ssl_context = OpenSSL::SSL::SSLContext.new
ssl_socket = OpenSSL::SSL::SSLSocket.new(socket, ssl_context)
ssl_socket.connect
session_id = ""
master_key = ""
ssl_socket.session.to_text.each_line do |line|
if match = line.match(/Session-ID\s*: (?<session_id>.*)/)
session_id = match[:session_id]
end
if match = line.match(/Master-Key\s*: (?<master_key>.*)/)
master_key = match[:master_key]
end
end
data = "GET / HTTP/1.1\r\n"
data << "Host: #{host}\r\n"
data << "Random: Foo/#{rand(36**10).to_s(36)}\r\n"
data << "\r\n"
ssl_socket.puts(data)
puts "Request:"
puts data
puts "----------- Response: #{ssl_socket.gets()}"
ssl_socket.close
File.open("keys_dump_ruby", "a") do |file|
file.write("RSA Session-ID:#{session_id} Master-Key:#{master_key}\n")
end
Here we'll run this and capture the traffic with Wireshark or tcpdump:
$ ruby ssl_key_export_session-ID_only.rb
Request:
GET / HTTP/1.1
Host: 192.168.1.214
Random: Foo/cl5dx8et9r
----------- Response: HTTP/1.1 200 OK
$ cat keys_dump_ruby
RSA Session-ID:DA6DDA7502BE043CE9ACAD482C994CCEF9282F6CAE7DCF48BC7314952E9D1F6B Master-Key:89C9F116F9C4778C234100858EFC05CC5E1F58D155E5A94CD10D4CF08D4473341019F1115B95820C2C3C7EE311BD5D5D
As expected, the traffic appears encrypted when opening the pcap with Wireshark:
Here we'll configure the key log file path in the SSL preferences and look at the packets again:
A new tab appears at the bottom of the window ("Decrypted SSL data"), which shows the data in the Packet Bytes section.
You can also display the full decrypted SSL session by right clicking on an SSL frame and selecting "Follow SSL Stream":
This is awesome so far, but we can increase the awesomeness by telling Wireshark which dissectors should be used to parse the decrypted data. For that, you need to use the first field of the SSL preferences: "RSA keys list." As we saw, this is the field where you list the server private keys to decrypt communications using the RSA key exchange algorithm. We are going to use it anyway to create an association and inform Wireshark which protocol needs to be displayed. I know, it is a little odd, but it is the only way I found to make it work.
Click on Edit, New and fill the form with the remote host IP address, the port number and the protocol to use (HTTP in this case). You also need to provide a private key in PEM format. Any key will do the job; it just has to be a private key in the right format. I have no idea why it has been designed this way, but it won't work without it. You can use the openssl command to generate a fake key, and it will work just fine: openssl genrsa -out private.pem
Now our packets are correctly parsed:
Export Client Random in Ruby file
This method works great, but sometimes the server may not return a Session ID. Usually, the server being configured to use Session Tickets is to blame:
As can be seen, the Session ID length is 0 in the Server Hello handshake message, and Wireshark won't be able to find the correct Master Key.
Each time a new session is initiated, both the client and the server create a random value as defined in RFC 5246. The client sends this value during the first step of the SSL handshake (Client Hello). Since this value is unique and generated for each session, Wireshark can also use it to identify the Master Key in the key log file:
Unfortunately, the Ruby binding for OpenSSL does not expose this value to the end user and, unless we patch the native code (a topic for another post), it requires a little bit of work to retrieve the Client Random. I used ffi-pcap, a Ruby FFI binding for libpcap, to extract the value directly from a network capture in the Ruby code.
Here is the how I did that (I just added the changes in the code of the last section for simplicity):
...[SNIP]...
require 'ffi/pcap'
...[SNIP]...
pcap = FFI::PCap::Live.new(:device => "en0",
:timeout => 1,
:handler => nil)
pcap.setfilter("dst host 192.168.1.214 and tcp dst port 443 and tcp[13]&8!=0 and tcp[32]==22 and tcp[37]==1")
pcap.non_blocking = true
ssl_socket.connect # 1) do |this, pkt|
pkt.body[77,32].each_byte { |byte| client_random << "%02X" % byte.ord }
end
...[SNIP]...
ssl_socket.close
pcap.close
File.open("keys_dump_ruby", "a") do |file|
file.write("CLIENT_RANDOM #{client_random} #{master_key}\n")
file.write("RSA Session-ID:#{session_id} Master-Key:#{master_key}\n")
end
The pcap filter will only capture outbound packets on port 443/TCP (dst host 192.168.1.214 and tcp dst port 443) with the PSH flag (tcp[13]&8!=0), SSL Handshake Content type (tcp[32]==22) and Client Hello Handshake Type (tcp[37]==1).
After calling OpenSSL#connect, the #dispatch block will extract the 32-byte Client Random (RFC5246 A.6) directly from the captured packet.
Let's try it now:
$ ruby ./ssl_key_export_session-ID_and_client_random.rb
Request:
GET / HTTP/1.1
Host: 192.168.1.214
Random: Foo/wet78yp4zs
----------- Response: HTTP/1.1 200 OK
$ cat keys_dump_ruby
CLIENT_RANDOM 38AB4EBBDE18601C0558EF9F8E8B00CBFB745D1F6BD01AA08A433F5C9E87AD22 12EA2924180E97C94CE8729A0AB178EE95EE8AEBADEC96A42A2016AAA7D3C330E46B37178137D3FFFA2118A295475A6D
RSA Session-ID:7C0DD8780BF5445B03A35BEEF47AD3F612FD6373F3002BED2F5766F01314923C Master-Key:12EA2924180E97C94CE8729A0AB178EE95EE8AEBADEC96A42A2016AAA7D3C330E46B37178137D3FFFA2118A295475A6D
We are now able to decrypt the SSL communication again:
Bonus: Monkey patching Ruby binding of OpenSSL
As a bonus, you can monkey patch the OpenSSL#connect method to be used with any Ruby code that uses OpenSSL.
require 'openssl'
require 'ffi/pcap'
module SSLKeyExport
def connect
pcap = FFI::PCap::Live.new(:device => "en0",
:timeout => 1,
:handler => nil)
# filtering on only tcp packets with PSH flag, SSL Handshake Content type and the Client Hello Handshake Type
pcap.setfilter("dst host 192.168.1.214 and tcp dst port 443 and tcp[13]&8!=0 and tcp[32]==22 and tcp[37]==1")
pcap.non_blocking = true
super
client_random = ""
pcap.dispatch(:count => 1) do |this, pkt|
pkt.body[77,32].each_byte { |byte| client_random << "%02X" % byte.ord }
end
pcap.close
session_id = ""
master_key = ""
self.session.to_text.each_line do |line|
if match = line.match(/Session-ID\s*: (?<session_id>.*)/)
session_id = match[:session_id]
end
if match = line.match(/Master-Key\s*: (?<master_key>.*)/)
master_key = match[:master_key]
end
end
File.open("keys_dump_ruby", "a") do |file|
file.write("CLIENT_RANDOM #{client_random} #{master_key}\n") unless client_random.empty?
file.write("RSA Session-ID:#{session_id} Master-Key:#{master_key}\n") unless session_id.empty?
end
end
end
class OpenSSL::SSL::SSLSocket
prepend SSLKeyExport
end
You just need to require this code using "-r":
ruby -r ./ssl_key_export_monkey_patch ./ssl_http_get.rb
Parting Thoughts
Thanks to this great Wireshark feature, you can now decrypt the encrypted network traffic generated by your Ruby code. This technique can be easily adapted to other languages too. Another approach would be to create a library that hooks the low level OpenSSL functions directly and dump the Session ID, Client Random and Master Key to a file. You would need to load your library before OpenSSL is loaded by using the LD_PRELOAD environment variable for example. Here is an implementation of that technique: https://git.lekensteyn.nl/peter/wireshark-notes/tree/src/sslkeylog.c.
