It has been ten years since the release of MS08-067. Unlike many of the other incidents over the years, this vulnerability has developed a celebrity life of its own (even including pillow shams!). It has a warm place in the hearts of the responders who worked that case too. Building on John Lambert's reflection, we have gathered a retrospective look back at the incident from the eyes of people on the front line of this response. For those of you looking for a quick refresher on the vulnerability that later became infamous behind Conficker take a look at our refresher deck here.
Introducing the responders…
Dustin Childs, Security Program Manager @ MSRC in 2008 – now with the Trend Micro Zero Day Initiative.
Ziv Mador, Senior Program Manager @ MMPC in 2008 – now Vice President of Security Research at Spiderlabs Trustwave.
Phillip Misner, Senior Security Program Manager @ MSRC in 2008 – now Principal Security Group Manager @ MSRC.
A Retrospective, 10 Years Later
Question: What role did you play in this release?
Dustin: In October of 2008, I was a security program manager in the Microsoft Security Response Center (MSRC). It was my job to coordinate the response to vulnerabilities affecting the Windows OS, meaning that among other things, I drove Windows bulletins. The MSRC case that eventually became MS08-067 was assigned to me. I had only shipped 11 bulletins total at this time, and none had been released Out-of-Band (OOB). This was obviously the most severe case I had ever been assigned, and I had no idea what all I had just signed up to do. I was also the person responsible for writing the actual bulletin. At the time, this was considered the #6 Worst Job in Science (according to Popular Science). They called us Microsoft security grunts, but I preferred the title of Redmond security gnome.
Ziv: Back in 2008, I was a senior program manager on the Microsoft Malware Protection Center (MMPC). One of my responsibilities was representing the antimalware team in discussions around emergency cases where we used the Software Security Incident Response Process (SSIRP). I worked closely with the MSRC team and with the malware analysts on fully understanding the details of vulnerabilities and potential exploits, developing detection rules and collecting telemetry for assessing the magnitude and spread of malware. The MS08-067 case, including its consequent Conficker variants, has been the most intense case we worked for and it lasted several months.
Phillip: At the time, I was the SSIRP Crisis Lead responsible for mobilizing and leading the response to the active attacks we observed. In that role, I was responsible for driving the risk understanding and engineering plan as well as coordinating the overall response.
Question: What are your recollections from being part of this response?
Dustin: It was amazing to see various pieces of the company come together to work on the response. We had teams in Redmond, India, and Europe all working pretty much around the clock. One thing I specifically recall is a moment during the first SSIRP meeting for this bug. There were more 15 people in the room and even more on the phone. When Phillip explained the bug, there was a brief moment of silence as we all immediately recognized a worm was on the way. From that point on, we knew were on the clock. People who didn't live through it might not understand it now, but that room was full of infosec professionals who had lived through things like Melissa, Nimda, Slammer, Sasser, and Code Red. That was another interesting thing about the response. I never had to convince anyone to drop what they were doing to work on this – I just had to explain what the vulnerability was. I had more conversations about whether to ship Moderate-rated cases over this one. Everyone involved understood the stakes. At the time, it felt like 17 days of everyone in Microsoft working on the same thing. The other big thing that stands out is the breadth of platforms impacted. Everything from Windows NT, which was still under custom support, through Windows 7 pre-beta was impacted. There were even calls for us to release a patch for Windows ME and 98, which were affected but long out of support. I heard we even had to re-spin DVDs so that Steve Ballmer wasn't handing out vulnerable copies of Win7 beta at the Build conference, but I never could confirm that one.
Ziv: It all started when the Trustworthy Computing folks, headed by John Lambert, approached us with the alarming information about a new severe zero-day in Windows. That team had developed a method for identifying unknown zero-days from crash reports. Using that method, they tracked the number of crashes from unstable attempts for exploiting the MS08-067 vulnerability. Once notified, the MMPC malware analysts developed an internal antivirus signature to help us find exploits of this dangerous bug. At that point, despite the huge malware repository the team maintained, very few non-replicating malware samples exploiting this bug were discovered. Later, in coordination with the release of the famous MS08-067 security update, we released the public signature to help protect customers and collect telemetry that would indicate the spread and geo-distribution of attacks.
We knew more malware would soon follow now that the information was in the public domain. The bad guys couldn't miss an opportunity such as this to exploit computers en masse during that brief time when the patch was not yet broadly installed. I published a blog on the MMPC blog urging people to install the patch with no delay. The title of that blog conveyed that message: "Get Protected, Now!"
As expected, this out-of-band release immediately captured the attention of the media, including Brian Krebs who was still writing for the Washington Post:
In the coming days, we held our collective breath, checking telemetry closely – almost hour-by-hour. More crashes were being reported, so it was obvious that people out there were experimenting liberally with those exploits. In early November, about a month after the initial discovery of the vulnerability, new malware emerged, exploiting it, however its prevalence was still very low. A few weeks later, the Conficker worm broke out but we'll keep that story to the second blog in this series.
Phillip: This was the first of several incidents where we were able to see attacks in early stages before they became widespread. I remember the excitement that we might have a chance to stop a much-broader attack. It was that opportunity that focused our conversations and drove the teams hard to develop a solution for this vulnerability. Seventeen days from knowledge to release with teams around the world working to secure customers was an amazing pace and experience to be part of. The second big memory of this incident was the spike in activity which led to the out-of-band release decision. As we worked diligently on the fix detections of this attack were one or two a day – clearly not a widespread attack. Then on October 15th we saw telemetry shoot up to twelve attacks. Admittedly twelve is not a big number, but it was a big jump in activity and we only saw the cases where it failed. For me at the time, that was a big jump in risk. The difference between the attacks we knew about and an internet-wide worm like Slammer or Blaster was just the good intentions of the attacker. That would be a phrase that I repeated as we debated our release plans and pressed the fixes. Ultimately we decided that the balance of risk favored a quicker release. It was only after the update release when we were able to retrieve logs from the attackers' infrastructure that we saw the spike we observed was a significant ramp and preparation for larger scale attacks (that combined graph is in the refresher deck we linked to above). The third big moment from this incident was the announcement of the release. The relatively new Advanced Notification Summary practice was executed 24 hours in advance of release. It was a short five sentences and it caught the industry off guard. Out-of-band releases were very rare and often associated with attacks in the wild. The anti-malware vendors were all scrambling looking for the attacks which up until that point only Microsoft had observed. The technology and telemetry we utilized had given us a crucial insight before wide-spread attacks occurred. I suspect we also caught the attacker by surprise. Once the update was released and the attacker's infrastructure identified, the original attacker simply walked away from their exploits. It was gratifying to be able to get this release out before that wide-spread damage could be done.
Question: What do you think went well in the response to the attacks addressed by MS08-067?
Dustin: To me, two things really stand out. First, go look at the bulletin and tell me what revision number it is. I'll wait. That's right – it still sits at version 1.0. Once we released it, we never had to go back and touch it more. The fix itself was generated really quickly. The rest of the time was spent testing the fix to ensure it really addressed the problem without breaking anything else on the system. More than any other case I worked, and I worked a lot of them, this was the one case where I felt the full weight of Microsoft working together to protect customers. The other thing we did well was tell the story. It was normal to do a webcast for the regular, Update Tuesday bulletin release. For this one bulletin, we did three – each one answering more than 60 questions from people watching. We worked with our PR and marketing partners to let people know how important it was to install this update. We even put a notification on msn.com. We did everything possible to let people know this was one patch that should not be ignored. I think that's reflected in how Conficker evolved. The first version, Conficker.A, showed up in under a month but caused very little damage. That version only used the Server service exploit to propagate. The later Conficker.B added the NetBIOS vector and became the monster we know today.
Ziv: I had been on the Microsoft Malware Protection Center for more than five years when the MS08-067 incident took place and through that journey, I saw immense improvement in the way the company and the community functioned and responded to such events. The entire response process was mature and included many teams at Microsoft that owned some part in the overall response: the Microsoft Security Response Center, the Trustworthy Computing group, the Microsoft Malware Protection Center, the customer support teams, the Windows product team, security communications, field, and government-outreach people and many more. The emergency response process included clear representatives from the participating teams, and they were mobilized based on need on very short notice. SSIRP war meetings were efficient and often convened in a matter of 15 minutes.
Innovation significantly contributed to the success. The method which John Lambert's team developed for identifying potential unknown zero-days from millions of crash reports was unparalleled. It allowed Microsoft to identify this dangerous vulnerability very early on before it was broadly used and afforded us the opportunity to issue a security update for Windows customers to install and protect themselves.
With respect to the team I was part of, the MMPC, the underlying infrastructure had improved significantly over the course of the years and supported our needs well. Enormous amounts of malicious and suspicious files were collected, scanned and stored in our systems every day, allowing us to quickly identify fresh malware as part of the outbreak, analyze it and get the necessary technical information, such as what vulnerabilities it exploits, how it spreads and what malicious servers or URLs it connects to. In certain instances, malware samples were provided to us by other security vendors and organizations, furthering our ability to stop the spread of the outbreak. The MMPC utilized comprehensive telemetry from hundreds of millions of computers globally. The most popular tool we used for collecting telemetry was the MSRT, but we also benefited from other security products of the time such as OneCare and the Forefront family of security products. This intelligence allowed me, as a response coordinator, to assess the magnitude of the outbreak and its global spread. That, along with crash data and input from customer support and the field, helped the company determine how to respond to the event strategically.
Phillip: Looking back at the response there are two items I feel we did exceptionally well at: speed of response and getting the right risk data to varied audiences. I mentioned before the world-wide effort to get this update produced – that was literal. At the time Windows Service Engineering was geographically split between our corporate campus (Redmond, WA) and the India Development Center (Hyderabad, India). The engineering teams did an exceptional job at utilizing resources following the sun and did not miss a beat in the development or testing of the update. The dedication and hard work those teams put in was remarkable. I tip my hat to my colleagues for all that they did to better protect customers. The second piece was getting that story and the risk data out. Being 10 years removed, I forgot how much effort we put into that response. There were new communications vehicles like the Advanced Notification Service, Microsoft Active Protections Program (MAPP), and our blogs that helped define what security response at Microsoft would be like for years to come. That day we upheld the principles of trust and protecting customers. Those were values from Trustworthy Computing and the lessons Microsoft had learned the hard way in the early 2000s.
Question: What lesson you wished we had learned or learned better as a result of this incident?
Dustin: The lesson I learned is how important it is to communicate the actual risk to customers. Security updates are often seen as obtrusive or a nuisance. Some have no faith in them at all. But there's ample evidence showing the best way to prevent compromise is to apply the available security updates. We as an industry need to be better about producing quality updates and talking openly and frankly about the risks they mitigate.
Ziv: Communication about the bug and the malware internally and, more importantly, externally, was critical for providing information for many teams and organizations. A series of bulletins, blogs, and articles from Microsoft were published and they were viewed by hundreds of thousands of people. It was clear that many security practitioners and teams needed reliable information and guidance during those several months and, by getting that out on a variety of channels, they were able to consume that information in a number of ways and mitigate their risk.
Phillip: If there was one lesson I wish we had learned as an ecosystem in 2008 it would be the need for systems to mitigate or patch our entire ecosystem. As I wrote the blog entries for WannaCry and Petya in 2017, I feel like we were in a similar place and conversation – only nine years later. The industry has evolved in capability (technologically and response-wise) and in our footprint of connected systems. Still, it feels like we have not advanced in the area to update or protect systems. I hope in the years to come when another attack appears that we as an ecosystem are better prepared.
Reflection on this incident
Today marks that 10 year anniversary from the release of MS08-067. When we started this response we did not think that it would have the lasting impact on the ecosystem, industry response processes, or our careers that it has. This represents a moment and some more insight into those days. A big thank you goes out to John Lambert for his first recollections published a couple years ago. It helped us to realize this story still has interest and relevance today as we look at an even more complex threat landscape.
Question: Dustin, any last thoughts on this anniversary of the incident?
Dustin: So that's it. I'm not sure what the "pinnacle of my Microsoft career" was, but this bulletin and the subsequent response was definitely a top 5 moment. I was proud to be a part of the team that worked on this, and I still tell people that MS08-067 is "my" bulletin.
Ziv: The MS08-067 was a unique experience where innovation, dedication and coordinated efforts all came together with the purpose of helping protect customers from forthcoming attacks. Those involved share a strong sense of pride for the role they played during the crisis.
We thank you for reading and hope some of the experiences prove insightful as new threats emerge for the ecosystem to tackle. In the next blog, we will discuss the outbreak of the Conficker worm, which exploited the MS08-067 vulnerability and affected many organizations out there. Expect that blog on the anniversary of that worm…