Securing Continuous Integration Services


Over the last couple weeks, I've had the distinct privilegeto share some of my research surrounding continuous integration security. The presentation was dubbed "Attacking CloudServices w/ Source Code" and was presented at both SOURCE Boston 2013 andTHOTCON 0x4, where I discussed a bunch of fun things like:

  • Why I love Continuous Integration (CI) Services(especially hosted solutions)
  • My perspectives as an open-source developer(some happy,some sad)
  • What things could be possible if malicious codewas fed to CI services
  • A project I'm working on, called RottenApple, tohelp make things better

In this blog post I hope to capture some of the meat of thepresentation for those who could not attend and use this opportunity to announce the first public release ofRottenApple.

Basic Terminology

I guess the first big question (maybe for some) is what iscontinuous integration?

Continuous integration (to me) is a system or process that monitors source code repositories for changes. If a change is detected (aka: a developer commits new code) the project is checked out built and tested to ensure that the software project still works.

Unit-tests (or "specs") are often used by continuous integration services to verify that everything still works in a software project. If something breaks, a unit-test will ideally fail and the continuous integration system will bubble that failure back up to a developer so that the problem can be addressed as soon as possible while the changes are fresh.

So, in short, developers use continuous integration toregularly checkout, build and run unit-tests on software projects for qualitycontrol. Continuous integration is theentity performing the inspection and Unit-tests are what validate that each andevery piece of software project works as expected.

Why I love ContinuousIntegration

Let me start by saying I haven't always loved continuousintegration. In fact, at one point itwas sort of a PITA. When initiallyjoining a group of Spiders that did a lot of development, I used to cringeat the idea of making a change that would cause the build to fail. In the famous words of one of my co-workers,"this makes you a bad person".

Anyways, beyond work stuff and fast-forwarding a bit, I was doinga little bit of open-source development stuff for fun and once I grew to appreciatethe value of having continuous integration for my projects at work, I reallywanted to apply the same techniques to introduce more quality control in myopen-source "hobby" projects.

One of the challenging parts of doing development for fun isthat you usually don't have the same resources as you would if you weredeveloping a commercial project. Thismeans you usually have a budget of $0 and you're usually finding time to commitcode between lunch breaks, while on the train or while pretending to watch TV. Even though I knew continuous integration would be good for my projects, I didn't have the drive or the additional time to invest on it because it would mean less time writing code (what I really want to be doing).

Travis-CI to the Rescue

Thankfully, after a little while, someone introduced me to a service by the name of Travis-CI. Travis-CI is a hostedcontinuous integration service for the open-source community. It's really easy to set up, mimics some ofthe stuff I have for my work projects and it's FREE. Now, there are other providers like this outthere, but I have the most experience with Travis-CI, so I reference Travisprimarily throughout.

In addition to being easy to setup and free, Travis-CI allows me to test my Ruby projects using a variety of different versions including 2.0, 1.9.3, jruby, rbx, ree, etc. It also allows me to have a full build history for each change made to the project and the icing on the cake for me was that it would also build any submitted pull request and let me know if the proposed changes would break the project or not before I merge them into master.

A Healthy Dose ofInspiration

Around the same time that I was discovering all this fancyhosted-CI goodness like it was Christmas morning, I was also trolling the AlohaRuby Conference videos and came across a video of a presentation called"Hacking with Gems" by Ben Smith. Thebasic gist behind Ben's talk was describing some of evil things you could doinside a Ruby gem and what sorts of things he tried to social engineer people,both actively and passively, to install his not so savory gems.

One of the most interesting components of Ben's presentationwas that he had a number of business cards made that simply stated "gem installaloha-ruby-conf" and placed them all around the conference. It's important to note that in this was adeveloper conference and it's more than common place for people to install gemsto try them out without really taking a close look at everything the gem doesunder the hood. During the presentation,after describing all the evil things that someone could do with a simple geminstall process, Ben provided a list of developer names that had installed hisbenign gem just to drive the point home.

Ben's talk really got me thinking and eventually I came towonder whether all these things (or similar techniques) were possible withouthaving to rely on tricking a user or highly obfuscate my code. This thought process led me back tocontinuous integration services.

Attacking ContinuousIntegration Services

When utilizing continuous integration services to build Rubyprojects, it commonly boils down to executing a rake task either explicitly via"rake spec" or simply through running "rake" and having the default behavior torun the unit tests for a given project. However, one of the things I noticed here was that my unit-tests themselves,even though I'm using rspec, which has some specific syntax for definingunit-test stuff, are just plain old ruby. This got me to the idea, "what if I just added malicious code to myspecs?". The system would mostcertainly execute my code and I'd get malicious code running on the CI.

The first thing Idecided to do before I got too far down the rabbit hole was to build my own CIserver for testing purposes. I did thisbecause I really enjoy having a GitHub account and I do (as I mentioned above)love having a free continuous integration service building all my projects and I didn't want to spoil that. Ididn't want to piss anyone off and I didn't want to feel bad when I did thingsthat would be considered unethically hacking another organization. I ended up building my own CI setup withJenkins-CI, which is a widely popular open-source CI server used by a largenumber of organizations for building commercial software.

The setup was very basic where by I would push codeto a private GitHub repository and my CI server would poll that project for anychanges. When I push malicious code upto my private GitHub repository, it would trigger a build on my CI server andthen I would get that malicious code to execute on the CI. It's an extremely simple setup (in devcircles), but the nature of the situation means that the development of"exploits" for these environments, if you even want to call them that, is veryeasy.

Here are a couple example offensive-biased things that I talked about in the presentation:

  • Breaking out of the build root (accessingneighboring project source code)
  • Performing a port scan of the CI's locallyattached network (potential for pivoting behind the firewall)
  • Authenticating back to GitHub using R-RW keys(potentially trojan the project)
  • Popping a Reverse Shell (getting command-lineaccess to the CI)

Although, throughout this description of all the offensivethings you can do with Ruby projects (all of my examples) there is nothing tosay that you couldn't do these same techniques with building/testing any otherlanguage such as PHP, Python, Java, etc. With such a fundamental level of trust here, the options are onlylimited by the trust-levels given to the CI to perform its activities.



After realizing that the nuances of CI security weren't going to beeasily resolved for the masses (both hosted and self-hosted continuousintegration services), I decided to start a project on GitHub calledRottenApple with the hopes of at least moving us it into a better direction. The main idea behind RottenApple is that you buildthe project on your continuous integration environment and it will test it andlet you know where there are weaknesses. Ironically enough, I've chosen to stick with a Unit-test concept wherethe roles are a bit reversed in that the unit-tests are actually testing the CI(not the code as they traditionally would).

After thinking long and hard about this tool, I decided tomake it a bit more multi-purpose and implemented two separate, but related name-spaces in theRottenApple project; (1) RottenApple::Audit – for safely auditing a target CIenvironment and (2) RottenApple::Attack – for actively attacking a target CIenvironment. I'm hoping that by havingthese two polar opposites that this project will help meet the needs ofDevelopers, System Administrators, CI Providers (both hosted and internalinstallments) as well as Security Practitioners.

Below includes the current feature set of the project by name-space designation:

RottenApple::Audit has the following checks:

  • Is the root user is being to build projects?
  • Can malicious code steal your RubyGems API key?
  • Could malicious code pivot to private networks?
  • Can malicous code authenticate using your GitHub creds?
  • Could malicious code receive instructions from a remote party or exfiltrate data from your CI?
  • Can malicious code access other projects being built on the same server?
  • Can malicious code steal SSH private keys?

RottenApple::Attack has the following features:

  • Steal the RubyGems API key
  • Flush IP Tables (aka: drop firewall rules)
  • Install Software to aid in the attack process
  • Make an unauthorized commit to master
  • Perform an NMAP scan of a desired set to targets
  • Throw/Shovel a reverse shell to get command-line access to the CI/CD
  • Steal SSH private keys

I just recently published the source code for this project on GitHub, which can be found here. I hope that people check it out and find ituseful. Hopefully, with some good feedback and maybe a little help from the community I'llbe able to extend it to do more interesting things over time.

Parting Thoughts

Lastly, I just want to make myself as clear as possible thatI don't see the techniques that I've described above as "0-days". Continuous Integration services are open bydesign for a number of reasons, including making sure that they don't inhibitthe ability for developers to quickly have their code tested and validated forregressions. However, it's important tonote that trust relationships do exist on these systems and can be abused ifnot carefully assessed.

I think we should do more to test continuous integrationservices for weaknesses to ensure the trust we have imparted them with is not abused. I'm hoping that you checkoutthe RottenApple project, find it useful and send me a pull request.

PS – Here is the deck from the presentation I referenced above. Also, the videos of the attack demos referenced in the presentation can be found here and here.

Trustwave reserves the right to review all comments in the discussion below. Please note that for security and other reasons, we may not approve comments containing links.