In this homework assignment, you will be using John the Ripper to crack some hashed and salted passwords.

Introduction

Password hashing

In the real world, you should never store users’ passwords in plaintext. If your database is ever breached, an attacker can immediately turn around and use the passwords they got to login as any user they want. Even worse, people have a bad tendency to reuse passwords a lot, so not only can the attacker login as anyone, but they may be able to reuse those passwords to login to their victims' other accounts.

To solve this problem, we use password hashing algorithms, which convert passwords like hunter2 into nice strings of random bytes, like

V0edGK/GfSrNwzYCrbML4V/gvkNuNTfvn.Pt/LMSAf8

Before hashing the password, we usually salt it first. We then compute the hash as

password_hash = hash_algo(salt + password)

and then store the salt (in plaintext) next to the password. This provides an additional layer of security against rainbow tables.

Tangent: you will often hear password hashing algorithms being referred to as key derivation functions, or KDFs. That’s because the more general application for these algorithms is for generating secret keys for cryptographic purposes. This will be relevant to us again in Programming Assignment 2 :)

One difference between KDFs used for password hashing versus the types of KDFs typically used for generating keys (such as HKDF) is that password hashing algorithms are typically very expensive to compute. This makes it much more difficult to brute-force someone’s password (as we’ll do in this assignment!).

The shadow file

Modern Linux distributions use the Shadow Password Suite¹ to store users' passwords in the /etc/shadow file. Here’s an example of what a shadow file might look like:

root:*::0:::::
nobody:!::0:::::
alice:$1$AimWoRko$AQI09Nus.zWeIgRbga4wD1:19274:0:99999:7:::
bob:$6$6qSLjtAxoE30/j7T$Zoyklx/79/AKVdttaLQKO82MRZ9Ocy2UYFp2N703Ad6zbuyuaUrPUX9gygX0WgCibuSvjBBrdUJmDl42ihZuv/:19274:0:99999:7:::

Tangent: Somewhat confusingly, while the /etc/passwd file may contain passwords, it usually does not. This is because /etc/passwd usually contains general user information that you want everybody to be able to read, whereas the hashed passwords in /etc/shadow are sensitive and should not be world-readable.

By separating passwords into a different file, Linux can set permissions on /etc/passwd so that anybody can read that file, while usually only the root user may read /etc/shadow.

Each line of the shadow file contains nine fields separated by the colon character (:). You can see a full description of these fields in the man pages for /etc/shadow (man 5 shadow). For our purposes, only the first two fields are important; these fields contain a username and that user’s corresponding password. The username is self-explanatory; the password is generated using the crypt function (man 3 crypt) and formatted as

$<id>[$v=<version>][$<param>=<value>(,<param>=<value>)*][$<salt>[$<hash>]]

Here’s an example from the Password Hashing Competition:

$argon2id$v=19$m=65536,t=2,p=1$gZiV/M1gPc22ElAH/Jh1Hw$CWOrkoo7oJBQ/iyh7uJ0LO2aLEfrHwTWllSAxT0zRno

This hash was generated from the password hunter2. You can read this by splitting up the password on the occurrences of $:

• The hashing algorithm is argon2id.
• The algorithm version is v=19.
• Argon2 takes three parameters, a “memory cost” m, “time cost” t, and “parallelism cost” p, which can be tuned to make it more difficult to crack (but also more expensive to compute). In this case, the values of those parameters are m=65536, t=2, and p=1.
• The salt used is gZiV/M1gPc22ElAH/Jh1Hw (this is a string of random bytes encoded to text using a slightly customized variant of Base64).
• Finally, the hashed password is CWOrkoo7oJBQ/iyh7uJ0LO2aLEfrHwTWllSAxT0zRno (which is again encoded with Base64).

Tangent: you might notice in the shadow file provided earlier that there were some lines that looked nothing like this, e.g.

root:*::0:::::

Accounts like these are not allowed to log in using a password. It’s often convenient to include shadow entries like this for high-privilege accounts that you want to restrict access to as much as possible.

Lab structure

In this lab, you will be using John the Ripper, a password cracking tool that computes password hashes as quickly as possible to try and figure out a password. You will be given a VM in Virginia Cyber Range, where you will need to crack the passwords of various users inside that VM.

What to submit

For Problem 1, you should submit the amount of time required to crack each of speedy1 through speedy4’s passwords, and determine which password hash was the hardest to crack (and by how much).

For Problem 2, you should provide the passwords of each of the users crackme1 through crackme5, and any John the Ripper rules you used to crack their passwords.

Grading

For each problem, you will be graded on your ability to complete the assigned objectives utilizing various features of John the Ripper.

Problems

Before you start cracking passwords, you will want to use John the Ripper’s unshadow tool to combine the /etc/passwd and /etc/shadow files into a single file. If you run

$ sudo unshadow /etc/passwd /etc/shadow

you will see a combined file listing all of the users on your VM in Virginia Cyber Range along with their hashed passwords (note that you need to use sudo to run the command as root, which will allow you to read /etc/shadow).

For this assignment, we only care about the speedy1 - speedy4 and the crackme1 - crackme5 users. To see the lines that only correspond to these users, you can run

$ sudo unshadow /etc/passwd /etc/shadow | \
    grep -E "^(speedy|crackme)"

where ^(speedy|crackme) is a regular expression matching all lines that start with speedy or crackme.

You will want to save these users’ hashes to a file. The command

$ sudo unshadow /etc/passwd /etc/shadow | \
    grep -E "^(speedy|crackme)" > passwords.txt

will save their hashes to the file passwords.txt.

Problem 1: comparison of password hash speeds (4 points)

There are four users, speed1 through speedy4, who each share the same password, idunno. However, the password hashing algorithm that each of them uses is slightly different:

• speedy1 (uid=2001) uses MD5;
• speedy2 (uid=2002) uses SHA-512 with 1000 rounds;
• speedy3 (uid=2003) uses SHA-512 with 10000 rounds; and
• speedy4 (uid=2004) uses yescrypt, which is a hashing algorithm based on scrypt. This is the hashing algorithm that recent versions of Debian, Ubuntu, and Fedora use by default.

For each of these users, you should use the time command with john to see how much time it takes to crack each user’s password. For this problem I recommend using one of the RockYou wordlists, in particular

/usr/share/seclists/Passwords/Leaked-Databases/rockyou-40.txt

Here is how you would run John the Ripper to crack speedy1’s password:

$ WORDLIST=/usr/share/seclists/Passwords/Leaked-Databases/rockyou-40.txt
$ time john --wordlist="${WORDLIST}" --format=crypt \
    --user=speedy1 passwords.txt

Repeat this command for the other speedy users. The output of time should look something like

$ time john [arguments...]
...
real    0.51s
user    1.60s
sys     0.09s

For each password you should record the “user” time. This is the amount of time that the CPU spent running that command.

Deliverables: report the amount of time that it took to crack each password. Which password hashing algorithm took the longest time?

Hints

Whenever John the Ripper correctly identifies a password, it saves that password to the file ~/.john/john.pot in the user’s home directory. This means that if you crack a password, and then try to run John the Ripper again on it, john will simply look in ~/.john/john.pot for the password that it already extracted and print that out.

This can be problematic if you need to re-run your tests to see how much time it takes for John to crack a password. Fortunately, there’s a simple solution:

$ rm ~/.john/john.pot

These deletes the john.pot file so that JtR doesn’t read cracked passwords from it.

Problem 2: password cracking with custom rules

In the second problem, you’ll need to use John the Ripper to crack the passwords of users crackme1 through crackme5. Each user has a different password taken from the wordlist

/usr/share/seclists/Passwords/Leaked-Databases/rockyou-75.txt

with the following modifications:

• User crackme1 (uid=3001) has appended !? to the end of their password (e.g. hunter2!?).
• User crackme2 (uid=3002) has wrapped their password with flag{ and } (e.g. flag{hunter2}).
• User crackme3 (uid=3003) has added four random bits to the end of their password (characters in {0, 1}) (e.g. hunter20101).
• User crackme4 (uid=3004) has added $Season$Year to the front of their password and ! to the end of their password for some $Season in {Spring, Summer, Fall, Winter} and some $Year such that 2020 <= $Year <= 2022. For example, the password hunter2 might become Fall2022hunter2!.
• User crackme5 (uid=3005) has capitalized the first letter of their password and prepended a month in {January, February, ..., December}. For example: hunter2 => AugustHunter2

For this problem, you will want to write some custom JtR rules to try new passwords. To do this, you should edit the file ~/.john/john.conf and add new rules at the bottom of the file; see the appendix for more information.

Deliverables: submit the passwords for each of these users, as well as the custom rules you defined for each password. You will get one point per password that you correctly identify.

Hints

• See the appendix for a bunch of different example John the Ripper rules; you’ll want to use these to design your own rules.

• You can see a list of all of the passwords that John the Ripper has cracked so far with

$ john --show ./passwords.txt

• You can give the --progress-every=... flag to have JtR print a progress report while it’s cracking passwords. For instance, --progress-every=3 would print out what percentage of the way JtR is through the wordlist every 3 seconds.

• To check whether a password is correct for a user, you can run su $username, e.g. su crackme3. If you enter the correct password you should get the message This account is not currently available., otherwise su will print out Authentication failure.

Appendix: Custom John the Ripper rules

John the Ripper allows you to define custom rules for passwords to attempt while cracking users’ passwords. To use a ruleset, you would pass into the --rules=... flag to john, e.g.

$ john --wordlist=/path/to/wordlist --rules=append_number passwords.txt

To add a new ruleset, you will want to edit the JtR configuration file, which you can do with

$ nano ~/.john/john.conf

or with a different text editor like vim. You will want to add your rules to the end of this file. In nano, you can jump to the end of the file by pressing Ctrl + _ followed by Ctrl + V; in vim, you can do so by pressing G while in normal mode.

Here are some example rulesets, which you can use to start writing your custom rules for Problem 2:

# ~/.john/john.conf

# This rule appends each number from 0 to 9 to each line in the wordlist. E.g.,
# if the password 'hunter' appears in the wordlist, JtR will try
#
#   hunter0
#   hunter1
#   hunter2
#   ...
#   hunter9
#
# Note: AN = "insert string at position". For example,
#
#   A0 = "insert string at the start"
#   A3 = "insert string at character position 3"
#
# In addition, Az = "insert string at end"
[List.Rules:append_number]
Az"[0-9]"

# This rule both appends *and* prepends every number from 0-2 to each line of
# the wordlist. Now if our password was 'hunter', JtR would try
#
#   0hunter0
#   0hunter1
#   0hunter2
#   1hunter0
#   1hunter1
#   1hunter2
#   2hunter0
#   2hunter1
#   2hunter2
#
[List.Rules:append_and_prepend_number]
# NOTE: "[0-2]" = "all characters between '0' and '2'". You could also write
# this as
#
#   A0"[012]" Az"[012]"
#
A0"[0-2]" Az"[0-2]"

# You can include multiple rules in a single ruleset! The following ruleset has
# one rule that prepends "hello" and another rule that appends "world", so that
# the password "hunter" would become
#
#   hellohunter
#   hunterworld
#
[List.Rules:hello_world]
A0"hello"
Az"world"

# Here's a mix of more exotic rules that can be used to do all kinds of other
# things
[List.Rules:exotic]
# Convert the word to lowercase and reverse it, e.g. hUnTer -> retnuh
l r

# Capitalize the first letter of the word, e.g. hunter -> Hunter
c

# Capitalize the first letter of the word, and then duplicate it
# e.g. hunter -> HunterHunter
c d

For a full list of all of the different rules you can apply, check out JtR’s wordlist rules syntax.