While there are many aspects of network pen testers that sets the good testers apart from the bad, three of the critical aspects are time management, data management, and tool mastery. The Metasploit Framework, a tool that is part of many of the network penetration tester tool kits, has some sophisticated data management and interoperability capabilities that can help improve time management and data management. This post will discuss how to leverage these abilities using Python, Metasploit and the msfrpc Python module recently released by SpiderLabs.
This post is going to walk through using the msfrpc Python module to import data into Metasploit, search the data, and automate addition recon tasks so that more time can be spent with complex vulnerabilities instead of doing mapping and enumeration tasks. All of these examples are going to be using the BackTrack Linux distribution. To begin, we need to setup our environment by installing PostgreSQL and git, and then installing the msfrpc Python module. To begin, we install the required BackTrack packages and the PostgreSQL gem for Ruby:
#apt-get install postgresql libpq-dev git-core
#gem install pg
Next, we get the latest version of the msfrpc Python module from the SpiderLabs git repository:
#git clone git://github.com/SpiderLabs/msfrpc.git msfrpc
#cd msfrpc/python-msfrpc
#python setup.py install
Now that all of our software is installed, the next step is to configure PostgreSQL so that we can utilize it inside Metasploit. To create the database and users we need, we will create a text file with our SQL commands and save it so that we can use it for any new BackTrack instance we may need. Create the file createdb_sql.txt with the contents:
create database msf;create user msf with password 'msf123';grant all privileges on database msf to msf;
Next, we need to run these commands on the PostgreSQL instance. This must be done as the postgres user, so we execute the command:
# sudo -u postgres /usr/bin/psql < createdb_sql.txt CREATE DATABASECREATE ROLEGRANT
The three lines at the end should be seen to indicate that each step succeeded. Now everything is setup, so let's get to the fun.
Launching the MSFRPC server
Before any of our scripting will work, the msfrpc server must be started. I prefer to do this within the msfconsole so that I have the option of using scripts or the console itself. To automate both database connections and starting the msfrpc server inside Metasploit we can create a rc file, a file full of commands that we want Metasploit to run automatically. Let's create our file and name it setup.rc:
db_connect msf:msf123@127.0.0.1/msfload msgrpc Pass=abc123
The db_connect command connects to our database using the username msf and the password msf123. It connects to the PostgreSQL server on the loopback address and connects to the msf database. This command will automatically create our tables and other schema elements that Metasploit will need to work.
By loading the msgrpc plugin, the msgrpc server will be started. By default the password is random, so we pass the Pass option to it with our pre-defined password. Abc123 isn't a great password, so you should choose something that is more fitting. It will be the password that we use through the following examples, so if you choose your own you will need to substitute it through the following examples.
Next, we launch Metasploit using our rc file and verify that that everything loaded successfully:
#/pentest/exploits/framework/msfconsole -r setup.rc
If everything worked successfully, you should see a stream of tables being created, and finally :
[*] MSGRPC Service: 127.0.0.1:55552 [*] MSGRPC Username: msf[*] MSGRPC Password: abc123[*] Successfully loaded plugin: msgrpcmsf >
Our MSGRPC server is now started, listening on the loopback address on port 55552.
Populating The Database
Our next step is getting information about our target hosts into the database. As part of the mapping process, we frequently use Nmap to map out hosts, ports, and services on the network. Nmap has the ability to print output to the screen as well as log to files. We can use the XML files that Nmap generates to populate our Metasploit database with the host information that we are looking for. For our examples we want to find SNMP servers that might be on the network. To do this, we are going to launch an Nmap scan looking for UDP ports.
The network we are using for this example is the 192.168.1.0/24 network. To launch an Nmap UDP scan and log to a variety of file outputs we are going to execute the command:
nmap -sU -oA nmap-udp 192.168.1.0/24
This will generate a number of files including .nmap files with the typical nmap output, .gnmap output which allows for easy grep, and .xml files which contain the XML files we are looking for. There are a number of ways to import the XML into Metasploit, but for this example we will be using Python.
To begin, we need to generate some basic Python that will read in the contents of our file. We want to be able to run our script on any compatible import file and have it imported into Metasploit. Let's write the code to read the file contents into a variable that we can use and put it in a file called import_file.py :
#!/usr/bin/env pythonimport sysimport msfrpcif len(sys.argv) < 2: print "Usage %s \r\n" % sys.argv[0] sys.exit()try: data = open(sys.argv[1],'r').read()except: print "File does not exist or cannot be read\r\n"; sys.exit()
This code makes sure that our msfrpc module and the sys module are both loaded. It then checks to make sure that we have specified a file to import when we run the script, and if we did not then it will print usage information. Next we try to read the file contents into a variable called data, and if it we can't read the data in throw an error.
Our next step is to write our code to connect to Metasploit, and authenticate with the system:
# Create a new instance of the Msfrpc client with the default optionsclient = msfrpc.Msfrpc({})# Login to the msfmsg server using the password "abc123"client.login('msf','abc123')
Here we are creating a new client instance. If we don't specify any options, it will use the defaults that Metasploit uses when it starts the server. Before we can execute any commands, we have to authenticate to the Metasploit instance, to do this we use the login function and specify username and password. If this fails, the module will throw an error.
The next step is to import the data. To do this, we have to use Metasploits database API. Let's look at the code:
res = client.call('db.import_data',[{'data' : data }])if res.has_key("result") and res['result'] == "success": print "Import successful"else: print "Import failed"
The import_data function in the database API will allow us to specify data to be added to the database. The call method will execute the first argument, which is the Metasploit API method to call. The second argument to call is an array with additional options to pass to the API method. The import_data function takes one argument, a dict containing options. In this case, we need to specify the data from the import file in the data value of the dict.
The call function returns a dict structure. The contents of the structure are different depending on the function called, but in this situation, if the data was successfully imported, the result value in the ret object will be "success".
To verify that all of the data really was imported, we can go back into the Metasploit console window, and issue the command hosts and verify that the hosts from our scan are listed.
Querying The Database
Once we have data into the database, many of the tasks we might want to do require getting data back out of the database. We're going to use the msfrpc Python module to do this. For our first task, we're going to look for all of the hosts that are in the database that have UDP port 161 open. UDP port 161 is the SNMP port. Let's start out a new script called snmp_scan.py by copying over our code that will connect to Metasploit:
#!/usr/bin/env pythonimport sysimport msfrpcimport time# Create a new instance of the Msfrpc client with the default optionsclient = msfrpc.Msfrpc({})# Login to the msfmsg server using the password "abc123"client.login('msf','abc123')
We've used the same code as before, with one minor addition, the inclusion of the time module. For some of the next steps we will need that module, so we've gone ahead and included it.
The next step is going to be to get the list of hosts that have UDP port 161 listening. To do this, we are going to use the db.get_service method in the Metasploit API. Let's look at the code:
params = { 'proto' : 'udp', 'port' : 161, 'state' : 'open' }res = client.call('db.get_service',[params])hosts = []try: for h in res['service']: hosts.append(h['host'])except: print "No services found\r\n"; sys.exit()host_list = ",".join(hosts)
First, we create a dict that contains our list of parameters we want to search for. In this case, we want to search for hosts that have port 161 open using the protocol UDP. We assign these values to the params variable. Next we call db.get_service with the params dict as our only argument.
The method returns a dict with all of the services found in an array assigned to the 'services' value. We first initialize an array to store our hosts that are found. Next, we iterate through each line of the services, containing information about each instance that was found. By looking at the hosts value of each dict, we can get the IP address of each instance found of the service. We add this into our hosts array. Finally, we join that list together into a single string separated by commas into the host_list variable.
If you want to see what's in this variable at this point, you can print out the host_list variable in the script and run a test to verify it works.
Executing Commands and Logging Output
Now that we have data, we can use that data as input to other commands. In this case, we already have a list of hosts with SNMP running, so now the logical step from a penetration testing scenario is to check these hosts to determine if they have easily guessable community strings. Metasploit has an auxiliary module designed to check for default community strings and will then verify if the community is read only or read/write.
The auxiliary module that we are going to use is the scanner/snmp/snmp_login module. It requires one variable be set, the RHOSTS variable which contains a list of comma delimited hosts to scan. When we run our scans, we want to be able to save the data to review as well as to use in documentation and screen shots. Even though we are doing some of these tasks through scripting, we want to be able to document the output that would be seen if these commands were being run manually.
Metasploit's RPC API contains a console class that allows us to create virtual consoles to run commands in. These virtual consoles allow us to submit commands, and read the data as if it were being input in our main console window. To create a new console, we add the following code to our script:
try: res = client.call('console.create') console_id = res['id']except: print "Console create failed\r\n" sys.exit()
The console.create method will create a new virtual console and return a dict with the id, prompt, and some other information about the virtual console. We need to save the console id to use later, so we save it to the console_id variable. We have this wrapped in error handling in case the console can't be created for some reason, in which case the script will exit out.
Next, we want to use the host list that we generated earlier to run a SNMP login scan. First we need to generate a string with our command in it, and then we need to write it to the console. We would do this just as if we were typing it in manually. To do this, we add this code to our script:
cmd = """use auxiliary/scanner/snmp/snmp_loginset RHOSTS %srun """ % host_listclient.call('console.write',[console_id,cmd])
Here we have created a command that will use the module, set RHOSTS to our host list, and then run the auxiliary module. We then call the console.write method with two arguments in our array, the first being the console id that we received from the console creation earlier, as well as the cmd variable containing our list of commands to execute. Now that we have launched our command, we want to watch for results. In our case, we're going to print this to the screen. We add this code to our script to read the output from our virtual console:
time.sleep(1)while True: res = client.call('console.read',[console_id]) if len(res['data']) > 1: print res['data'], if res['busy'] == True: time.sleep(1) continue breakclient.call('console.destroy',[console_id])
The first thing we want to do is to wait for one second to ensure the command has started. Sometimes it takes a second or two from the time we submit an action until it happens in the console. Next we create a while loop that will continue to read data from the console until our command is finished running. We use the console.read method to read data from our console. The method requires one argument, the console id of our console.
Our loop will go forever until we explicitly tell it that it can stop. Each iteration, we check to see if there is any new data. the console.read method returns a dict with two critical pieces of information: the data written so far and the busy status of the console. If there is data then it will be printed to the screen. While the console is busy, we will sleep for one second for additional work to happen before we read again. If the console isn't busy, then we can break out of our loop.
Finally, once we are done with our console, we should destroy it to limit memory. The console.destroy method will destroy our virtual console. It takes only one argument, the console id to destroy.
Trying It Out
Let's try importing a file, and trying our scan script. The Nmap data file we will be using is called nmap-demo.xml. Our commands would be:
root@bt:~/scripts# ./import_file.py nmap-demo.xml Import successfulroot@bt:~/scripts# ./snmp_scan.py | tee snmp-out IIIIII dTb.dTb _.---._ II 4' v 'B .'"".'/|`.""'. II 6. .P : .' / | `. : II 'T;. .;P' '.' / | `.' II 'T; ;P' `. / | .'IIIIII 'YvP' `-.__|__.-'I love shells --egypt =[ metasploit v4.2.0-dev [core:4.2 api:1.0]+ -- --=[ 786 exploits - 426 auxiliary - 128 post+ -- --=[ 238 payloads - 27 encoders - 8 nops =[ svn r14544 updated today (2012.01.12)RHOSTS => 192.168.1.110[*] 192.168.1.110:161 - SNMP - Trying public...[+] SNMP: 192.168.1.110 community string: 'public' info: 'Linux bt 2.6.39.4 #1 SMP Thu Aug 18 13:38:02 NZST 2011 i686'[*] 192.168.1.110:161 - SNMP - Trying private...[*] 192.168.1.110:161 - SNMP - Trying 0...[*] 192.168.1.110:161 - SNMP - Trying 0392a0...[*] 192.168.1.110:161 - SNMP - Trying 1234...[*] 192.168.1.110:161 - SNMP - Trying 2read...[*] 192.168.1.110:161 - SNMP - Trying 4changes...[*] 192.168.1.110:161 - SNMP - Trying ANYCOM...[*] 192.168.1.110:161 - SNMP - Trying Admin...[*] 192.168.1.110:161 - SNMP - Trying C0de...[*] 192.168.1.110:161 - SNMP - Trying CISCO...[+] SNMP: 192.168.1.110 community string: 'private' info: 'Linux bt 2.6.39.4 #1 SMP Thu Aug 18 13:38:02 NZST 2011 i686'[*] 192.168.1.110:161 - SNMP - Trying CR52401...[*] 192.168.1.110:161 - SNMP - Trying IBM...[*] 192.168.1.110:161 - SNMP - Trying ILMI...[*] 192.168.1.110:161 - SNMP - Trying world...[*] 192.168.1.110:161 - SNMP - Trying write...[*] 192.168.1.110:161 - SNMP - Trying xyzzy...[*] 192.168.1.110:161 - SNMP - Trying yellow...[*] Validating scan results from 1 hosts...[*] Host 192.168.1.110 provides READ-WRITE access with community 'private'[*] Host 192.168.1.110 provides READ-ONLY access with community 'public'[*] Scanned 1 of 1 hosts (100% complete)[*] Auxiliary module execution completed
Here we can see that we have first imported our data file, and then the SNMP scan has been successfully run. We have piped our output through the tee command allowing output to both be printed to the screen and logged to a file so that we can use it in reports.
Wrapping Up
This was just a brief introduction to using the msfrpc Python module for Metasploit automation. If you liked this module, and are more of a Perl fan, there's also a Perl module distributed in the git repository for you to try. With these basics to get you started, there are already a lot of possibilities for areas where scripts like these can be used to optimize the pen testing process.
If you already know some scripting, but need a reference for Perl or Python, Coding for Penetration Testers can help with some of the basics for both languages. You should also check back here for more on other cool SpiderLabs projects.
