Trustwave SpiderLabs Uncovers Ov3r_Stealer Malware Spread via Phishing and Facebook Advertising. Learn More

Trustwave SpiderLabs Uncovers Ov3r_Stealer Malware Spread via Phishing and Facebook Advertising. Learn More

Managed Detection & Response

Eliminate active threats with 24/7 threat detection, investigation, and response.

Co-Managed SOC (SIEM)

Maximize your SIEM investment, stop alert fatigue, and enhance your team with hybrid security operations support.

Advisory & Diagnostics

Advance your cybersecurity program and get expert guidance where you need it most.

Penetration Testing

Test your physical locations and IT infrastructure to shore up weaknesses before exploitation.

Database Security

Prevent unauthorized access and exceed compliance requirements.

Email Security

Stop email threats others miss and secure your organization against the #1 ransomware attack vector.

Digital Forensics & Incident Response

Prepare for the inevitable with 24/7 global breach response in-region and available on-site.

Firewall & Technology Management

Mitigate risk of a cyberattack with 24/7 incident and health monitoring and the latest threat intelligence.

Microsoft Exchange Server Attacks
Stay protected against emerging threats
Rapidly Secure New Environments
Security for rapid response situations
Securing the Cloud
Safely navigate and stay protected
Securing the IoT Landscape
Test, monitor and secure network objects
Why Trustwave
About Us
Awards and Accolades
Trustwave SpiderLabs Team
Trustwave Fusion Security Operations Platform
Trustwave Security Colony
Technology Alliance Partners
Key alliances who align and support our ecosystem of security offerings
Trustwave PartnerOne Program
Join forces with Trustwave to protect against the most advance cybersecurity threats
SpiderLabs Blog

Username Enumeration against OpenSSH-SELinux with CVE-2015-3238

I recently disclosed a low-risk vulnerability in Linux-PAM versions prior to 1.2.1 which allows attackers to conduct username enumeration and denial of service attacks. The purpose of this post is to provide more technical details around this vulnerability.

The Past

Time-based username enumeration is an old topic, yet it keeps coming in and out of fashion.

OpenSSH has sort of always been affected by this issue due to bugs in PAM or OpenSSH itself. It is still affected today and unlikely to be patched (it doesn't seem to be on the OpenSSH developers' priority list).

Demonstration on Debian 7.8 fully patched:

$ ssh_login host= user=FILE0 0=logins.txt password=$(perl -e "print 'A'x50000") --max-retries 0 -x ignore:time=0-3 
16:28:56 patator INFO - Starting Patator v0.7-beta ( at 2015-04-02 16:28 AEDT
16:28:56 patator INFO -
16:28:56 patator INFO - code size time | candidate | num | mesg
16:28:56 patator INFO - ----------------------------------------------------------------------
16:29:19 patator INFO - 1 22 23.258 | john | 8 | Authentication failed.
16:29:22 patator INFO - 1 22 25.597 | root | 2 | Authentication failed.
16:29:22 patator INFO - 1 22 25.593 | joe | 5 | Authentication failed.

What happens is that when sending an overly long password, the server takes longer than usual to respond if the provided username is valid because the server goes through the costly operation of computing the password hash only if the user account exists on the system.

There is also the Metasploit module ssh_enumusers.rb but it is not multi-threaded (even with THREADS=10 it seems to only test one username at a time).

The Bug

Earlier this year, a fellow SpiderLabs colleague contacted me because this trick was not working against the OpenSSH server he was pen-testing. The server would always take the same time to respond, even for root.

This seemed strange so I went back to test this again but on a CentOS VM this time, instead of Debian, and indeed it wasn't working. I then tried with much longer passwords and found out that the server would take 2 minutes to respond for a valid username when the password was 65536 characters or more.

Demonstration on CentOS 7.1 fully patched (same results with CentOS 6.6):

$ ssh_login host= user=FILE0 0=logins.txt password=$(perl -e "print 'A'x65536") --max-retries 0 -x ignore:time=0-3 
16:30:13 patator INFO - Starting Patator v0.7-beta ( at 2015-04-02 16:30 AEDT
16:30:13 patator INFO -
16:30:13 patator INFO - code size time | candidate | num | mesg
16:30:13 patator INFO - -----------------------------------------------------------------------------
16:32:13 patator INFO - 1 22 119.968 | root | 2 | Authentication failed.
16:32:13 patator INFO - 1 22 119.927 | joe | 5 | Authentication failed.
16:32:13 patator INFO - 1 22 119.928 | john | 8 | Authentication failed.

The "fun" part was actually tracking down where the bug was buried.

The Source

When SELinux is enabled (by default on CentOS/RHEL), libpam actually uses the external program /sbin/unix_chkpwd to verify the provided password. Otherwise when SELinux is disabled, libpam computes the password hash itself and compares it against /etc/shadow (in which case we only need to send a password of a few thousand characters to notice a time difference).

Let's have a look at the source:

$ curl -s -o - | rpm2cpio | cpio --extract --to-stdout 'Linux-PAM-1.1.*' | tar xjf -
or for CentOS 6.6 but the code is the same

I've simplified the code below, but basically at some point we step into _unix_verify_password() which then calls get_pwd_hash():

// in modules/pam_unix/support.c:
int _unix_verify_password(const char *username, const char *password)
retval = get_pwd_hash(username, &shadowhash)
if (retval != PAM_SUCCESS) {
if (retval == PAM_UNIX_RUN_HELPER) {
retval = _unix_run_helper_binary(username, password);
} else {
retval = verify_pwd_hash(password, shadowhash);

Then get_pwd_hash() calls get_account_info():

// in modules/pam_unix/passverify.c:
int get_pwd_hash(const char *username, char **shadowhash)
retval = get_account_info(username, shadowhash);
if (retval != PAM_SUCESS) {
return retval;

In get_account_info() below, if SELinux is enabled then we go back to _unix_verify_password() above and end up calling _unix_run_helper_binary():

// in modules/pam_unix/support.c:
#define SELINUX_ENABLED is_selinux_enabled()>0 // true unless SELinux=disabled in /etc/selinux/config

int get_account_info(const char *username, char **shadowhash)
if (geteuid() || SELINUX_ENABLED) {

The _unix_run_helper_binary() function creates a pipe and forks. The child process calls execve() on /sbin/unix_chkpwd which will then read the password from the pipe (as stdin), while the parent writes the password to the pipe and waits for the child to return:

// in modules/pam_unix/support.c:
static int _unix_run_helper_binary(const char *passwd, const char *user)
int retval, child, fds[2];
if (pipe(fds) != 0) {
return PAM_AUTH_ERR;

child = fork();
if (child == 0) {
dup2(fds[0], STDIN_FILENO);
execve(CHKPWD_HELPER, {"/sbin/unix_chkpwd", user, NULL}, {NULL});
} else if (child > 0) {
if (write(fds[1], passwd, strlen(passwd)+1) == -1) {
retval = PAM_AUTH_ERR;
while (waitpid(child, &retval, 0) < 0);

On Linux, a pipe has a capacity of 65536 bytes by default (16 pages of 4K each) and so what happens is that the parent blocks on the write() call because it tries to write strlen(passwd)+1 bytes.

The child reads the password from the pipe and truncates it to 200 characters (see read_passwords() in modules/pam_unix/passverify.c and the #define MAXPASS 200). This is the reason why the SSH server takes the same time to respond whether you send a 1-character or a 50k-character password.

So the child computes the hash, compares it to the actual one in /etc/shadow and returns (i.e. the /sbin/unix_chkpwd process exits), while the parent remains blocked. Hence the child process is now a zombie and shows as "<defunct>" in ps. And we just hang like this until the server disconnects upon hitting LoginGraceTime which is 120 seconds by default.

Final Thoughts

Beware that if you use Patator's --timeout option or ssh_enumusers.rb's THRESHOLD option you will DoS the server if you hit too many valid usernames (sshd will temporarily deny new connections after hitting MaxStartups).

The recommendation for this type of issue is obviously that the application logic should take the same compute time whether the username is correct or not. However, Linux-PAM decided to just truncate the input password to 512 bytes.

Also worth noting is that some question whether unix_chkpwd follows best coding practices. So there could be other bugs.

Responsibly Disclosed

Bonus Tip

To enable debug logs in pam on CentOS:

# yum install yum-utils rpm-build -y
# wget
# rpm -ivh ./pam-1.1.8-12.el7.src.rpm
# yum-builddep rpmbuild/SPECS/pam.spec
# sed -i -e 's,^%configure ,%configure --enable-debug ,' rpmbuild/SPECS/pam.spec
# rpmbuild -bb rpmbuild/SPECS/pam.spec
# rpm -Uvh rpmbuild/RPMS/x86_64/pam-1.1.8-12.el7.centos.x86_64.rpm
# echo 'touch /var/run/pam-debug.log; chmod 622 /var/run/pam-debug.log' >> /etc/rc.d/rc.local
# reboot
(run `yum downgrade pam` to revert to original version)

You can reach me at @lanjelot for more intel. Thanks for reading!

Latest SpiderLabs Blogs

Hunting For Integer Overflows In Web Servers

Allow me to set the scene and start proceedings off with a definition of an integer overflow, according to Wikipedia:

Read More

Welcome to Adventures in Cybersecurity: The Defender Series

I’m happy to say I’m done chasing Microsoft certifications (AZ104/AZ500/SC100), and as a result, I’ve had the time to put some effort into a blog series that hopefully will entertain and inform you...

Read More

Trustwave SpiderLabs: Insights and Solutions to Defend Educational Institutions Against Cyber Threats

Security teams responsible for defending educational institutions at higher education and primary school levels often find themselves facing harsh lessons from threat actors who exploit the numerous...

Read More