Last Friday I was trying out some new code that one of mycolleagues wrote to help automate some of the work involved in releasing newversions of the TrustKeeper Scan engine. One of the many things the code did was send emails. I hate writingboilerplate emails, so I was excited to put it to use and save myself some precioustime. Unfortunately, when I ran the codefor the first time, it crashed with the following error when trying to connect to our Exchange Server...
Now, this error is pretty self-explanatory and having spenttime working with other Ruby libraries that utilize OpenSSL, this basicallymeans that we're failing to verify the certificate of the server we'reconnecting to. The interesting part tome, was that when I visit this URL with Chrome and other web browsers, theysuccessfully verify the certificate provided. Weird huh?
In this blog post, I'll explain some of the digging around Ihad to do to get to the bottom of this issue and some other interesting bits Ifound along the way.
A Gem, Inside a Gem, Inside a Gem, Inside a Gem
First of all, Ruby Gems are pretty cool because you can usethem as building blocks to build something bigger badder and meaner. One of the tricky aspects of having such astructure like this is tracking down who's responsible for an error when yourun into problems.
In our case, we were using the Ruby Viewpoint gem. The Viewpoint gem provides a thin layer ontop of Microsoft Exchange Web Service (EWS) and lets you do all kinds of funthings with Exchange, including sending emails. After getting the above error, I was able to track the failure down throughthe gem dependency chain down to it's source, which turned out to be just a couplegems deep.
- Layer 1: Viewpoint Gem - A light layer for talking Exchange Web Services
- Layer 2: Handsoap Gem - A library for creating SOAP clients in Ruby
- Layer 3: httpclient - A library for http protocol communication
- Layer 4: OpenSSL - A library that interfaces with native OpenSSL
Certificate Verification Nuances in Ruby OpenSSL
So to get right down to it, we're basically trying toestablish an SSL wrapped socket with the target service. We can do this quite easily using Ruby OpenSSL:
Once we execute this code, we'd expect to see the same erroras we saw above, right? Wrong, the codeexecutes just fine and we've successfully connected to the SSL wrappedsocket. I'll show you in an IRB session to prove it…
The reason the connection doesn't pop the verify exceptionis because by default verify mode on SSLContext objects are set to nil. This effectively means they are set to "OpenSSL::SSL::VERIFY_NONE"or more plainly no verification. This also means that any SSL connection building using the default SSL context are susceptable to MiTM (Man in the Middle Attacks), but anyways, ONWARD!
So if we explicitly set the Ruby OpenSSL verification modeto be more strict, we can can generate the same error that I saw when initiallytrying out that automated email code like so...
Now this is sort of good and sort of bad. If you remember earlier in this post, I notedthat Ruby OpenSSL was failing to verify the certificate, but then my browser(Chrome in my case) was actually verifying the certificate. So I'm still stuck.
As it turns out Ruby OpenSSL does not populate it's certstore by default. In fact, much likeits default verify mode, the default cert store is set to nil.
This means that if we want to verify a certificate, we needto tell OpenSSL where to go for this information. This also means that we need to create acertificate store and populate it with sufficient information from the system'scertificate store to verify the target servers provided certificate. This can be done like so...
Now that we've determined what could be the source of ourproblem, we can now start working our way back up the gem dependency stack tosolve this issue for ourselves.
Working Our Way Back Up the Stack
Now that we've got a solid understanding of why SSL wascomplaining, lets start with httpclient and it's usage of OpenSSL. If we open up the source code forhttpclient's HTTPClient implementation we can also see the SSLConfigimplementation, which creates an OpenSSL::X509::Store object oninstantiation, but does not populate the store using the systems cert storeusing the #set_default_paths method. Unfortunately enough, the library also sets it's default verificationlevel to SSL::VERIFY_PEER, which is a good default, but when mixed with anempty cert store, it's going to be a common error case for casual SSL userslike I've shown above.
However, there is a silver lining. If we look deeper intothe source file, we can see that the author provided a method off of theSSLConfig implementation #set_default_paths, which allows us to configureOpenSSL such that it will verify peers using the systems certificate store.
Our next stop is the handsoap gem. The handsoap gem implementshttpclient from within it's Handsoap::Http::Drivers::HttpClientDriverclass. The SSL configuration there is very minimaland is carried by the SSL configuration options set on the request objects thatare passed into this class. The optionsavailable are to set a trust_ca_file, a client_cert_key_file and/or the sslverification mode. A configurationoption does not exist from within the handsoap interface to tell httpclientthat he should use the system cert store.
Our current state leaves most users with a similarincentive, as we saw with some of the httpclient SSL defaults, which would be todisable SSL verification. Fortunatelyfor us, there is an alternative since we're in the wonderful land of Ruby andit's called Monkey Patching.
Monkey Patching Our Way Out of A Jam
So, we're in a use case where the existing library (handsoapin our case) doesn't provide the ability to configure httpclient's sslconfiguration options based on what we need. Well, in Ruby there is a concept called Monkey Patching that gives us away out of this jam.
I'm sure there are much more formal descriptions andexplanations of Monkey Patching, but my layman description is that we canoverride the existing implementation's class dynamically to add the featuresthat we're looking for.
In the case of our Viewpoint usage, I can copy the handsoap'sHttpClientDriver class implementation into a separate file, make the changes I want and make sure it getsloaded after the Viewpoint gem. Thiseffectively reloads the HttpClientDriver class and causes the underlyinghandsoap library to use it rather than its implementation of the same class. Here are the modifications to the file we'll call http_client_driver_path.rb:
All we need to do is get the load order such that we loadthe vanilla implementation and then load our custom Monkey Patched version ofthe class like so:
The way Monkey Patching works in this case is sort of like whoever gets the last word in on the definition of an Object wins. So by loading our patch after the fact, we effectively can trick the handsoap gem being used by Viewpoint into using our implementation rather than theirs. This allows us to send emails with ease, continue to use SSL verification and not have to deal with those pesky exceptions that put a wrinkle in my Friday Afternoon.
This is also a really useful technique for exposing interfaces on Ruby implementations for security testing and such, but we'll save that discussion for another day.
I realize this was sort of a random run down half rant of some of mytroubleshooting activities from last Friday, but I still think there were somegood lessons learned in the process and I'll try to summarize them here:
- RubyOpenSSL SSLContext objects have a default verification setting of nil, whichappears to be equivalent to VERIFY_NONE (no verification of SSL certificates).
- RubyOpenSSL SSLContext objects have a default cert store setting of nil, whichappears to be equivalent to an empty store (certificate validation will fail ifverify mode is set to VERIFY_PEER).
- Rubyhttpclient is using VERIFY_PEER as it's default verification mode, but has anempty cert store by default.
- Rubyhandsoap has minimal cert_store configuration control causing many casual SSLusers to disable verification when they run into the above SSL certificatefailure problem.
- MonkeyPatching, although sometimes controversial, can be aneffective way of working around restrictive interfaces that incentivize casual SSL usersto disable verification.
I also did a little bit of unscientific digging around onGitHub to see how many projects are disabling certificate verification for Ruby OpenSSL using VERIFY_NONE. As it turnsout, it's pretty common and I got about 27,000 hits in my query. I've provided the query here in case you'reinterested in seeing which projects have certificate verification disabled.
GitHub Search for Ruby OpenSSL VERIFY_NONE projects
Additionally, I'd like to point out that if you are using an application that defaults to Ruby OpenSSL (or just OpenSSL in general) VERIFY_NONE, you run the risk of making the application suseptable to MiTM (Man in the Middle) attacks. If you're using Ruby-based projects, checkout their source code and see how your code or the code you use deals with certificate verification and certificate stores. You might be surprised.
Thanks for reading. I'll see you next time!