The exploitation of the CitrixBleed vulnerability in Netscale by a variety of ransomware groups has led to a widespread disruption of services across several industry sectors, including financial services, healthcare and real estate.
Dozens of companies are now trying to recover from these attacks, with some being unable to conduct operations due to the severity of the attack. The other reason could be they did not have a good incident response and recovery plan in place.
These organizations could have avoided this situation if they had taken the time to conduct tabletop exercises. While these exercises would not reveal any specific vulnerabilities, they would have given each firm hands-on practice to react properly when confronted with a worst-case scenario.
All of the victimized companies were or are still dealing with the aftermath of the attack, and the training regime that was in place prior to the attack likely decided whether they experienced a short, smooth recovery or one that is quite lengthy and painful.
Tabletop Exercises Help Relieve the Pain
Trustwave has conducted hundreds and hundreds of tabletop exercises with financial institutions, healthcare providers, and even the UK Ministry of Defense. While each exercise is different and centered on a specific type of attack, the results are generally the same.
The organization participating is better prepared and practiced when security incidents occur.
So, let's continue looking at this through a CitrixBleed lens.
A proper tabletop exercise is well-planned, with the client deciding what type of attack should be the focus. Let's say ransomware.
Trustwave, working alongside the client, will create a playbook that will run the client's team through a series of situations they would face while trying to recover from an actual ransomware attack.
While the client's leadership knows the exercise is planned, little or no warning is given to those tasked with handling the problem. In some cases, we tell them to appear in a conference room to conduct a security drill.
When everyone is gathered, we brief the client's team on the situation and say, "Here's who we are, here's what we're doing, and what we hope to accomplish."
We open up the scenario at a particular point. For this discussion, let's say IT has discovered the attack, has remediated or contained it, and is starting the recovery phase.
The first response from the client that we would expect to see, but seldom do, is to have the team pull out their recovery manual, and initiate their process. Our team does not allow the process to go smoothly because it never does in real life, so we start mucking up their operation by asking questions and throwing in "gotcha" problems.
For example, the attackers are demanding a $5 million ransom payment. Do you want to, or can you pay it?
Paying the ransom is always a great question, and we have found the client is often not prepared to answer what is almost always the first problem arising from a ransomware attack. Then we start applying more pressure: do you have the cash on hand? Do you need to go to the board for approval? Do you want to bring in law enforcement?
That line of questioning in only one aspect of what is taking place. On the technical side, we'll say that the database has failed.
What do you do? What will happen? The database has five different arms, so now five other applications are impacted; what is the proper course of action?
Do you have enough backup storage to hold your data during the restoration process? We recently conducted one such test, and the client only had enough to hold 95% of its data, then it failed, and they could not recover.
These problems are not inserted into the exercise to be cruel or show any fault with the client's team. We apply pressure to see how the team reacts and how the dominoes fall. In the end the client leaves the exercise with a lessons learned exercise to identify what went well, and what they can improve on for the next time.
Expanded Tabletop Exercises
Generally, tabletop exercises just include the technical teams, but an ideal situation is when we can simultaneously run an executive tabletop. In these events, the technical team is in one room, and the C-Suite members, legal, public relations, and other necessary parties are in another.
In this situation we might see the CISO running between rooms trying to answer questions from both groups and lead the recovery. We will toss in issues for the executive team to handle, like telling the PR manager that news of the attack in the press, what will you do? We present other issues for legal, bring up the ransom payment issue with the company's leaders.
The underlying point we are testing is if the client's recovery manual is effective.
One of the first words I want to hear when we kick off a program is, "Let's grab the manual." Without a manual directing operations, you end up with five techs, each running in their own direction, and that only creates chaos.
Everyone has to be on the same page, and everyone must know the correct action to take at each step in the recovery process, or it could make a bad situation much worse.
For example, the first line in the manual should read, "Don't unplug the system." You can disconnect from the infected network, but if power is removed, the forensic information needed to identify and prosecute the threat actor is gone.
In the End, Who Wins?
A tabletop exercise is not meant to have a winner and a loser. Trustwave does not go in with any pre-set goals the client's team must meet. We want to see how the team performs, process what happens, and then make recommendations so the client will do better the next time, or during an actual attack.
To be effective, the client must conduct these exercises regularly to build institutional muscle memory. This way, the people responding don't simply pull the power plug and think they have saved the day.
Instead of winning, we want to uncover issues. Does the client's team have the skills to handle an attack, and if not, what skills need to be added? Has the client backed up its data?
Interestingly, this is the number one fault we find with most clients. Unreliable backups.
In retrospect, maybe there is a goal to be reached. That is, to expose a client's weaknesses beforehand so they are stronger down the road.
