If your text file is 16GB in size and you only have 2GB of free RAM, the command will fail. awk needs to load the entire megawordlist_wpa2 file in to memory in order to sort and determine which lines are duplicates. In this example, megawordslist_wpa2_nd would contain all of the same entries as megawordlist_wpa2, but with all of the duplicates consolidated to a single line in each instance, resulting in zero duplicate lines. awk '!seen++' megawordlist_wpa2 > megawordlist_wpa2_nd If you have duplicates, there’s a quick and easy command that can get rid of them all.
There may be none (in which case you can skip to the next section), or there may be tens, or hundreds, or thousands of duplicate lines. sort megawordlist_wpa2 | uniq -dĭepending on the size of the file, this command will take a long time to execute, but it will print any and all lines that are duplicates. Here’s a quick command that we can run to make sure. There’s nothing wrong when doing this, but what if we want to make sure that there are no duplicate entries in, for example, megawordlist_wpa2 from above. cat wordlist1 wordlist2 wordlist3 > megawordlist You can easily concatenate multiple text files together with the following command, which is great for creating a large wordlist from many small wordlists. When combining multiple lists, this can result in a lot of wasted time.
Removing Duplicate Entries from a WordlistĪ lot of wordlists have very common passwords that will also appear on other wordlists.
We’re left with a file called megawordslist_wpa2 that we can use to crack WPA2-PSK hashes with no wasted time. grep -x '.\' megawordlist > megawordlist_wpa2 This syntax uses a megawordlist file for example, and outputs a new file which has only passwords that range in length from 8 to 63 characters, which comply with WPA2-PSK requirements.
The following command uses WPA2-PSK as an example for length, and cuts all lines out of a password wordlist file that don’t meet the requirements for a WPA2-PSK password. Why would we bother with 63 character passwords? Again, waste of time. The requirements for a WPA2-PSK password is that it be at least 8 characters, but no more than 63 characters. We don’t want to bother trying the 1-7 character passwords, because that’s a waste of time.Īnother scenario – say you’re cracking WPA2 pre-shared key passwords. That’s fine, but what if we have a list of hashes and we know that their network policy requires that the passwords be a minimum of 8 characters in length. It may contain passwords anywhere from 1 characters all the way up to 20+ characters. For example, you may have a giant, multi-gigabyte password list that days hours and hours to run. The first thing that we may want to do to drastically reduce the time it takes to crack passwords is to cut down a wordlist based on the requirements of the password that we’re attempting to crack.
Modifying Wordlist Lengths with the ‘grep’ Command When concatenating many wordlists together, you’ll run in to two issues specifically, and those are duplicate entries, and password lengths based on known rules or policies. Because of this, we want to keep wordlists as concise as possible. When cracking a password, each and every password that’s in a wordlist takes time to hash, compare against the password hash, and then either be successful or (most likely) throw away and continue on. You may have concatenated a bunch of wordlists together to create one giant wordlist for long, unattended cracking sessions, but there may be some problems when doing that.
In a couple of previous articles ( here and here), we learned several things to consider regarding the collection and combination of password wordlists, and how to use the cat command to combine multiple wordlists in to one. Password Cracking: Pairing Down Wordlists via the Linux Command Line