Measuring password re-use empirically

February 9th, 2011 at 19:11 UTC by Joseph Bonneau

In the aftermath of Anonymous’ revenge hacking of HBGary over the weekend, some enterprising hackers used one of the stolen credentials and some social engineering to gain root access at rootkit.com, which has been down for a few days since. There isn’t much novel about the hack but the dump of rootkit.com’s SQL databases provides another password dataset for research, though an order of magnitude smaller than the Gawker dataset with just 81,000 hashed passwords.

More interestingly, due to the close proximity of the hacks, we can compare the passwords associated with email addresses registered at both Gawker and rootkit.com. This gives an interesting data point on the widely known problem of password re-use. This new data seems to indicate a significantly higher re-use rate than the few previously published estimates.

A simple intersection yielded 522 email addresses registered at both sites. This is about a 1% overlap, small but reasonable given the very different niches of the two websites. Eliminating throwaway addresses from sites like Mailinator and dubious addresses like spam@spam.com (it’s not clear that either site properly checked the validity of enrolled emails) left 456 pairs.

Analysing password re-use requires inverting the hashed passwords since the sites used different hash algorithms (and Gawker minimally salted their hashes). Rootkit.com’s password implementation is worse than Gawker’s, with no salts at all and just a single iteration of MD5, meaning it’s quick to test a huge dictionary of known passwords. I cracked 44% of the accounts using a dictionary of about 10 M entries in less than 5 minutes. I previously used the same dictionary on the Gawker dataset and cracked 54% of the accounts (despite this, the passwords at rootkit.com were generally weaker, with many more being from a smaller list of common passwords).

Of the 456 common users, 161 had their password cracked in both datasets, 46 only had their rootkit.com password cracked and 77 only had their Gawker password cracked, leaving 172 with neither password cracked. Of the accounts for which passwords were cracked at both sites, 76% used the exact same password. A further 6% used passwords differing by only capitalisation or a small suffix (e.g. ‘password’ and ‘password1′). Some of these were due to the use of crypt() at Gawker, which truncated longer passwords to 8 characters. The remainder appeared to use unrelated passwords and I saw no site-specific password tailoring such as ‘gawker-password’ and ‘rootkit-password’.

This isn’t an accurate estimate, however, because none of the users whose password was cracked at only one site could have reused the same password (since the same dictionary was used). Including these numbers, the apparent re-use rate is only 43%. If we include the similar passwords, and assume that 6% of the passwords cracked at one site but not the other were also similar but one variation was not in our dictionary, we would estimate 49% of users employed very similar passwords between the two sites.

This still isn’t quite a complete comparison because we’ve ignore the 172 users with neither password cracked. We might assume that a roughly similar proportion of these users reused their passwords. It’s likely though that these more security-conscious users had a lower re-use rate, meaning 49% is an over-estimate. Still, we have to estimate at least a 31% re-use rate even if none of this last group of users reused the same password.

Either rate is much higher than what we would estimate based on the best published studies – Flôrencio and Herley’s empirical study (about 12%) or Gaw and Felten’s user survey (about 20%). Sampling error due to random chance shouldn’t be more than about ±5%, which can’t explain the difference. It could be that users are much more likely to reuse a password between Gawker and rootkit.com, since both protect access to forums and are of relatively low value. It could also indicate that password re-use has risen significantly in the past 5 years (which Gaw and Felten specifically predicted based on their survey).

More data is clearly needed because the difference between a 10% re-use rate and a 50% re-use rate would change the economics of large-scale attacks. It would also be very interesting to study the password overlap between higher-value accounts, such as those with a large email provider or an online bank, with low-security accounts like Gawker and rootkit.com which are more likely to be compromised.

Entry filed under: Authentication, Security engineering, Security psychology, Web security

14 comments Add your own

  • 1. Tyler Moore  |  February 9th, 2011 at 19:55 UTC

    Fascinating analysis. One potential issue that you didn’t mention is the question of the representativeness of the sample. Gawker attracts a fairly broad demographic with its different blogs, but presumably rootkit.com attracts a geekier crowd, so users of both services are also likely more technical than average. This makes the result even more surprising, since you would expect technical users to have lower rates of password re-use.

    One final thought — I wonder whether the segment of the population that actually comments on blogs and forums is representative of the wider Internet population. LBT excluded, I very rarely post comments on blogs, and I would expect that most Internet users don’t either.

  • 2. David Malone  |  February 9th, 2011 at 21:23 UTC

    Hi Joseph ,

    I have a writeup that looks at reuse between a few password data sets that I’ve collected. A draft is at:

    http://www.hamilton.ie/dwmalone/PasswordReport.pdf

    Section 5 has some statistics that are relate to reuse.

    David.

  • 3. dazzlepod  |  February 10th, 2011 at 17:40 UTC

    Cleartext rootkit.com passwords published at http://dazzlepod.com/rootkit/.
    Indeed many passwords appear to be reused @ twitter, facebook, gmail, etc. Reminder to users to keep password unique to the site where it is used.

    I emailed Jussi without response so far whether rootkit.com will send email notice to their users of the breach and change their password elsewhere.

  • 4. Clive Robinson  |  February 10th, 2011 at 19:55 UTC

    I suspect that if the stats become available (unlikley considering the implication) that you will find several basic classes of user.

    1, uses the same password for everything
    2, uses the same base password with pre/post numbers etc
    3, uses the same password for sites at the same level but has multiple levels (so fred1234 for no-impact sites like nonpay newspapers through to complex passwords for their bank etc)
    4, Those who use a password safe or password construction program such that every sight has a different nearly random password and a single master password known only to the user..

    Of the latter group expect over confidence and thus weak security question usage for password reset…

    There will of course be atleast one more group,

    5, Users whos method remains unknown but use unique passwords for the sites in the study group.

  • 5. Anon  |  February 11th, 2011 at 15:13 UTC

    Due to the crowd attracted to rootkit.com, I wouldnt get excited about finding passwords like “123456″ and “password123″. Most guys will be well aware that the site will probably get compromised and used a “throw away” password for sure.

  • 6. Matthew Slyman  |  February 15th, 2011 at 16:54 UTC

    I have devised a method of strongly discouraging password reuse. I would welcome comments on the validity or vulnerability of my approach.

    http://www.slyman.org/blog/2011/02/preventing-password-reuse/

  • 7. Curious George  |  February 17th, 2011 at 11:19 UTC

    How are you able to determine what type of hash is used for the passwords (MD5) and whether or not it has salt, etc? A 16-byte hash doesn’t necessarily mean MD5.

  • 8. Joseph Bonneau  |  February 17th, 2011 at 11:59 UTC

    @Curious George: To be perfectly accurate you can’t rule out the use of a different, unknown hash function. But the evidence that MD5 is in use is overwhelming because all of the common passwords (password, qwerty, 123456) etc yield MD5 hashes (including after truncation) which are in the database. If another, known hash like SHA-1 were in use, then somehow all of the users have picked passwords never before seen elsewhere, since no common passwords yield a SHA hash in this database. Alternately some other unknown hash could be in use which happens to yield many outputs which are also MD5 hashes of common passwords. This is probabilistically impossible unless the designers were intentionally trying to do fool people looking at the database, and I sincerely doubt they were.

  • 9. Curious George  |  February 17th, 2011 at 16:25 UTC

    @Joseph Bonneau,
    Thanks for the response. I was just wondering how you or the original hackers made the determination of the type of hash used since I was doubtful that the database contained that indication. It also leads to my next question of how to determine if salt and/or iterative hashing were used with the hashes and if so, how to identify and duplicate it.

  • 10. mark seiden  |  February 17th, 2011 at 23:59 UTC

    i am a bit confused: if the rootkit.com passwords were unsalted md5 why most of them need to be cracked at all versus simple lookup in rainbow tables. surely there are precomputed md5 rainbow tables for a keyspace large enough to include all of your weak passwords, based on my search query
    rainbow tables md5 hashes…

  • 11. Agent X  |  February 18th, 2011 at 17:48 UTC

    Agree with @anon. There wasnt much on site that would require high-protected password vs commercial Gawker.
    Also you are missing dormant or people not chancing passwords for years. That is different than reuse since time makes bruting affordable.

  • 12. lazystoner  |  February 21st, 2011 at 22:58 UTC

    What you’ve got to remember is that the majority of coders have become lazy, with too many websites to login to and too much copy and pasting, it’s easy to get sloppy…

  • 13. Roger  |  February 28th, 2011 at 10:41 UTC

    @Clive:
    Your list is interesting, but I think there is an important category missing:
    3a. has multiple levels, with different and increasingly strict policies at each level, ranging from reusing passwords like “fred1234″ for no-impact sites like nonpay newspapers, through more complex browser-managed passwords for blogs, through to Password Safe for genuinely important passwords like banking.

    I base this class on the fact that I do it, and know of at least two other people who do so, neither of whom is particular interested in computer security.

    I based my policy on “limiting the attack envelope” (i.e. a remote intruder can’t attack my Password Safe when it’s on a thumb drive locked in the physical steel safe) but the other two were just balancing security against convenience. And impressively well, I thought.

    @Joseph Bonneau:
    Since you have email addresses for many of these persons, I wonder would you consider contacting them fir a survey about the reason their password was reused?

    It would need to be diplomatically worded, since they may already be angry, and may tend to assume you are a phisher.

  • 14. Joseph Bonneau  |  February 28th, 2011 at 11:29 UTC

    @Roger

    It wouldn’t be appropriate to send unsolicited email to the tens of thousands of people affected by incidents like this-their email addresses were not given out voluntarily so this would be an unqarranted re-use of the data. I also would guess that, practically, the response rate would be so low as to seriously harm the validity of the survey, because sampling bias would be significant.

Leave a Comment

Required

Required, hidden

Some HTML allowed:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Subscribe to the comments via RSS Feed


Calendar

February 2011
M T W T F S S
« Jan   Mar »
 123456
78910111213
14151617181920
21222324252627
28