Cracking a “shadow” password using John the Ripper

In this short article, I’ll walk you through the steps of cracking a password stored in the /etc/shadow file on a Linux machine. Keep in mind that in order to access the shadow and passwd files, you need root access.

Step 1:

Extract the user’s entry from the passwd file and the shadow file and put them in text files for John the ripper (replace the USERNAME with the username of your choice):

sudo cat /etc/passwd | grep USERNAME > passwd.txt
sudo cat /etc/shadow | grep USERNAME > shadow.txt

Step 2:

Use the unshadow tool that is part of John the ripper tool set to create a single text file that contains both entries of the user into on line:

unshadow passwd.txt shadow.txt > unshadow.txt

The resulting file would be a combination of the user’s entries from passwd and shadow. This step organizaes the data needed by John in a format that John recognizes.

Step 3:

Choose a dictionary of possible passwords, such as Kali’s rockyou.txt (contains over 14 million passwords), and run John:

john --wordlist=/usr/share/wordlists/rockyou.txt unshadow.txt

If the password is found within the given wordlist, you’d see the output like this:

password (USERNAME)

Step 4:

If you get the famous “No password hashes loaded”, then the cryptographic hashing algorithm used is not easily recognized by John.

Take a look at the unshadow.txt file. The field after the username (with a number or letter between two dollar signs) is the one that identifies the hash type used. It could be one of the following:

  1. $1$ is MD5
  2. $2a$ is Blowfish
  3. $2y$ is Blowfish
  4. $5$ is SHA-256
  5. $6$ is SHA-512
  6. $y$ is yescrypt

For $y$, for example, you can use the command:

john --format=crypt --wordlist=/usr/share/wordlists/rockyou.txt unshadow.txt

 

How to install MITRE ATT&CK Navigator locally

MITRE ATT&CK framework has become a de facto standard when working in threat hunting, incident response, among other areas in defensive security. The online navigator provided by MITRE can be useful in understanding the current state of the attack campaign, what to do next, and can also help in attack attribution. However, I don’t feel very comfortable using an “online” tool hosted by someone else in mapping an ongoing attack in an organization. You might be a bit old-school like me, and that’s what landed you on this page. This is only possible because MITRE has provided a version that you can host and use locally.

Make sure that you have Docker installed before you start. If you don’t have it installed, you can install it using the steps mentioned here.

The following steps show how to install and run the MITRE ATT&CK Navigator locally as a container on an Ubuntu machine:

git clone https://github.com/mitre-attack/attack-navigator.git
cd attack-navigator/nav-app/src/assets
wget https://raw.githubusercontent.com/mitre/cti/ATT%26CK-v14.1/enterprise-attack/enterprise-attack.json
wget https://raw.githubusercontent.com/mitre/cti/ATT%26CK-v14.1/mobile-attack/mobile-attack.json
wget https://raw.githubusercontent.com/mitre/cti/ATT%26CK-v14.1/ics-attack/ics-attack.json

The first command will download the whole repo, and the last three commands will download the required assets from the website so that your setup can get it locally, without contacting the server.

The next step is to change the configuration of the navigator to use the local assets instead of the internet-based ones. Edit the ~/attack-navigator/nav-app/src/assets/config.json file to change each of the “data” fields in the “ATT&CK v14” space to the local resources:

Change

"data": ["https://raw.githubusercontent.com/mitre/cti/ATT%26CK-v14.1/enterprise-attack/enterprise-attack.json"]

to:

"data": ["assets/enterprise-attack.json"]

And change:

"data": ["https://raw.githubusercontent.com/mitre/cti/ATT%26CK-v14.1/mobile-attack/mobile-attack.json"]

to:

"data": ["assets/mobile-attack.json"]

And finally change:

"data": ["https://raw.githubusercontent.com/mitre/cti/ATT%26CK-v14.1/ics-attack/ics-attack.json"]

to:

"data": ["assets/ics-attack.json"]

The next step is to go back to the ~/atttack-navigator folder, and create the new docker container:

cd ~/attack-navigator
sudo docker build -t attack_navigator .

Finally, we start the container locally:

sudo docker run -p 4200:4200 attack_navigator

Wait for a minute, or two, and then open your browser and go to http://127.0.0.1:4200

 

Last-URL – A command line python tool to reveal redirecting URLs

I have put together a super simple tool to extract the last URL from a redirecting URL. Threat actors utilize redirections to deliver malicious payloads or hide phishing URLs. In some scenarios, the attackers would use URL shortening tools (such as bit.ly) to bypass detection algorithms in emails, SMS messages, or on web pages.

The tool is not available on my github page on this link:

https://github.com/Mo-Alani/last-url

It can be used in the command line to show the last URL as shown in this example:

python last-url.py https://tinyurl.com/285mdnbr

Or to show the whole history of redirections (if the link redirects more than once):

python last-url.py -h https://tinyurl.com/285mdnbr

If the URL contains “&” sign, you’ll need to put the URL in quotations

python last-url.py -h "https://tinyurl.com/285mdnbr"

Let me know in the comments if you face any trouble using the tool, or you have suggestions for improvements.

Installing Volatility Framework on an Ubuntu Virtual Machine

Edit 19-Feb-2024: This article was written for Volatility 2 which was based on Python 2.x.

Although a bit old, Volatility Framework is still one of  the favourite tools for memory forensic investigations. Its wide range of plugins enables easy extraction, although without a fancy interface, of a lot of important pieces of information.

Today, we’ll walk through the process of installing volatility framework on an fresh installation of Ubuntu. Let’s kick this off:

Part 1: Creating a VM and installing Ubuntu

My personal favourite is VMWare Workstation Pro, but you can easily use VirtualBox to achieve the same goal.

1. Download Ubuntu 18.04LTS from this link:

https://releases.ubuntu.com/18.04.6/ubuntu-18.04.6-desktop-amd64.iso

2. Create a new VM with 4GB of RAM, and at least 20GB (I recommend 40GB to have some space for the memory dumps you’ll examine)

3. Create a username and a password of your choice. For my installation, I’ll use “mohammed” as my username.

4. Leave the rest of the installation settings to the default.

Part 2: Preparing the Python environment

Volatility Framework was written for python 2.x, not 3.x. This means that you’ll need to prepare an environment of python 2.x with all the dependencies.

1. Download Miniconda from this link:

https://docs.conda.io/en/latest/miniconda.html#linux-installers

2. Start the installation at the terminal window by moving to the ~/Downloads folder and issuing the following command:

sudo sh Miniconda3-py39_4.10.3-Linux-x86_64.sh

Change this “Miniconda3-py39_4.10.3-Linux-x86_64.sh” to the name of the file you just downloaded if it is different. During the installation, leave the installation location to the default, and respond with “yes” to “Do you wish the installer to initialize Miniconda3”.

3. After the installation is done, close the terminal window and open a new one. You should see the prompt (base) before your terminal prompt. Just like this:

4. To check the version of Python that is currently installed, type the command:

python --version

It should show you 3.9.5.

5. to be able to create a new conda environment, you will need to change the ownership of the .conda folder in your miniconda installation. This can be done by:

sudo chown mohammed:mohammed /home/mohammed/.conda

Replace all the “mohammed”s in this command with your selected username, including the folder name.

6. As Volatility Framework requires Python 2, not 3, we will create a new environment using the following commands:

conda create --name py2.7 python=2.7

7. After the creation of the new conda environment, we can activate it using the command:

conda activate py2.7

This will change the prompt to py2.7.

8. Now we will re-check the version of python using the command:

python --version

It should show 2.7.18.

9. Now we install the libraries needed by volatility using these commands:

sudo apt install pcregrep libpcre++-dev python-dev git gcc -y

pip install distorm3

pip install yara-python

pip install PyCrypto

pip install pillow

pip install OpenPyxl

pip install ujson

Part 3: Installing and Using Volatility Framework

1. Move to the ~/ folder using cd ~/ command.

2. Download the volatility framework using this command:

git clone https://github.com/volatilityfoundation/volatility.git

3. Change the folder to ~/volatility using the command cd volatility

4. Test the installation using the command:

python vol.py –info

5. Take a look at the different plugins and profiles. You’ll notice that the profiles included in the framework are all Windows profiles. The framework doesn’t include any Linux or Mac profiles by default. You’ll need to download these profiles from here:

https://github.com/volatilityfoundation/profiles

6. For Linux profiles, it is tricky to find the profile that fits your particular distribution and kernel versions. It is a common practice that you compile a profile from the machine where you’re capturing the memory dump using tools such as “Lime”.

7. A typical command to have the framework check automatically what is the most suitable profile would look like this:

python vol.py --file="/home/mohammed/Desktop/memdump.dump" imageinfo

After finding the suitable profile for your memory dump, you can issue commands like this:

python vol.py --file="/home/mohammed/Desktop/memdump.dump" --profile=Win7SP1x64 psscan

The psscan plugin would show you the processes running in memory at the time of the capture.

Handling IP Addresses in Machine Learning Datasets

As I was working on a small research project that employs a typical cybersecurity dataset, I came across this question. How should we handle IP addresses in these datasets? Here are my thoughts about this.

Many datasets used in research conducted in cybersecurity contain IP addresses, such as source IP address, and/or destination IP address. In most of these datasets, IPv4 addresses are expressed in the usual dotted decimal format (ex 192.168.0.1). Most programming languages, like Python, will go crazy when they see a number with three decimal points and hence the question; how do we handle them?

A question that I would like to pause before addressing the IP address handling problem; how useful are IP addresses in training models?

If you’re training a model to detect intrusions, what impact does the source IP address has on your model’s decision to classify the packet/stream as malicious? The source IP address of a single packet might not be of great help to recognize the packet as malicious. However, if that is combined with other packets maybe coming from the same source, there might be some value in knowing the source IP address. In a similar conclusion, the destination IP address of a single packet might not help in the proper classification process, but combined with packets coming from many other sources to the same destination might flag a DDoS attack. This, makes classical classification algorithms incapable of properly classifying traffic based on features relevant to a single packet.

There are other scenarios where some or all the IP addresses in the dataset do not really contribute to the classification process. On the contrary, these addresses can confuse the classifier and through the results into the wrong class. In a dataset that contains IP addresses

Handling them as a String (really??)

If you choose to handle them in your model as strings, then their actual contribution to the training of your model is close to nothing. I advise highly against this because it doesn’t help the classification process by any means.

Handling them as Decimal numbers

Handling IP addresses as numbers can take one of the following two paths:

Path 1

Convert the IP address to a binary numbers, and then convert the binary number to an integer. IP addresses can easily be converted into binary numbers from their original dotted-decimal format as each octet can be converted into an 8-bit binary representation. Then, you combine these 4 8-bit octets into a single 32-bit number, and covert that to a decimal number.

The code below shows a way of doing this in python:

Assuming that the IP addresses are held in a list named “array”,

for pnt in range(len(array)):
    x = array[pnt]
    z = 0
    parts = x.split('.')
    z = (int(parts[0]) << 24) + (int(parts[1]) << 16) + (int(parts[2]) << 8) + int(parts[3])
    array[pnt]= z

Keeping in mind that in most cases, the IP address itself is not an indicator of an attack, and most of the time it doesn’t hold a value in the detection process. However, as we explained earlier, the relationship between different IP addresses might help the classification process for some types of attacks. The relationship between IP addresses might indicate whether the attacker is an internal actor, or an external actor as well.  With that in mind, this method maintains this relationship between IP addresses, as the system would be able to learn that IP addresses that are close to each other can be within the same network. This can help in recognizing attacks coming from botnets within the same IP cluster, for example.

Path 2

In this path, we strip the dots from the dotted-decimal and simply add zeros to the left to complete each octet to exactly three digits. This means that the IP address 192.168.1.14 becomes 192168001014. This python code shows how this can be done

for pnt in range(len(array)):
    x = array[pnt, 0]
    z = ''
    parts = x.split('.')
    for prt in range(3)
        if len(parts[prt])=3:
            z = parts[prt] + z
        elif len(parts[prt]=2:
            z = '0'+ parts[prt] + z
        else:
            z = '00' + parts[prt] + z
        array[pnt]= int(z)

 

This method also maintains the relationship between IP addresses and helps the recognition of adjacent nodes and networks.

One Hot Encoding

In this method, IP addresses get encoded into representative codes instead of being dealt with as values (whether string, or integer ones). When using this method, each IP address would be represented as a code (such as a binary code). This way, you can identify each packet coming from, or going to, a particular IP address. However, this representation will eliminate the relationship between these IP addresses. This means that your system would not be able to identify the relationship between IP addresses of two hosts within the same network.

Although the relationship is not maintained between IP addresses in this representation, it can be helpful in recognizing data flows, and doing statistics analysis of data originating or landing in a specific node. Even if the system doesn’t know what the exact IP address in a packet, it can easily calculate the amount of traffic this node is sending or receiving, or the amount of data going from a particular source to a particular destination.

For example, if your dataset contains 110 unique IP addresses, these will be replaced with 7-bit codes. The first IP address appearing in the dataset would be name ‘0000000’. Consequently, and mention of the same IP address will be encoded as ‘0000000’. The second IP address, and all of it’s consequent mentions, would be replaced with ‘0000001’ and so on.

In python this can be done through the use of SKLearn Label Binarizer.

Final Remarks

As we discussed earlier, IP addresses are not always helpful to include within the training process. In any way you choose to represent them, they can hinder the system’s learning and have a damaging impact to the system’s outcome. A general rule of thumb is that do not include IP addresses in your training data unless you recognize a tangible benefit from it. If you’re implementing a system to detect denial of service attacks, IP addresses can help. If your system needs to differentiate internal from external traffic, you need to include IP addresses. Just look at how the IP address impacts the attack detection process and make a decision.

If you’ve used other ways of handling IP addresses, I’d love to hear from you. Perhaps, I’d include it in this article as an update. Let me know what you think in the comments section below, or shoot me an email.

Subnetting with Variable-Length Subnet Mask (VLSM)

In this tutorial, we will go through steps of subnetting a network address using multiple subnet masks. If you’re looking for classful subnetting, you checkout my other tutorial here. After the detailed steps, we will go through two completely solved example and three more unsolved examples for you to solve. Grab a pencil and a piece of paper and let’s go.

To start with, we will assume that you have a specific number of subnets with a specific number of hosts in each subnet. What we will do in the process of subnetting is identifying the subnet address, subnet mask, subnet broadcast address, range of available host addresses for each subnet.

Subnetting generally means using some bits of the host address to generate subnet address.

Subnetting Steps:

Step 1: Choosing the Network Address

If you were given a specific network address to use in subnetting, jump to step 2. If it is up to you to choose the network address and IP address class, you may select it based on the following steps:

  1. For each subnet calculate a BlockSize.
  2. The BlockSize is calculated by taking the No. of Hosts+2 and rounding it to the next power of 2 (ex: 2, 4, 8, 16, 32, 64..etc which is actually 21, 22, 23, 24, 25,..etc). When we say “round to the next”, it means that 17 is rounded to 32 and 5 is rounded to 8. The result of rounding is the BlockSize. Now we have the BlockSize for each subnet.

Note: We put +2 because the subnet address and the broadcast address cannot be used by hosts.

  1. Add all BlockSize for all subnets. If the sum is 256 or less, you can go for Class C address. If the result is higher than 28(which equals 256) but less than 216 (which equals 65,536), go for Class B. If the result is more than 216, go for Class A.
  2. Since subnetting mostly happens for LANs, the addresses that we will use are in the Private ranges of each class as shown in the table:
Address Class Private Range of IP addresses
C 192.168.0.0 – 192.168.255.255
B 172.16.0.0 – 172.16.255.255
A 10.0.0.0 – 10.255.255.255

For the sake of simplicity, we usually choose 192.168.0.0 for Class C, and 172.16.0.0 for Class B.

Step 2: Choosing the Subnet Mask for Each Subnet

The calculation is really simple. Using the BlockSize that you have calculated in Step 1, you can find the number of bits required for host address after subnetting (Lets call it HBits) by the following calculation:

BlockSize=2HBits

This means that only HBits number of bits will be used for host addresses and the rest of the bits will be used for subnet addresses.

From that, you can easily calculate the number of bits used for subnet address (lets call it NBits) by:

NBits=32-HBits

Now you can produce the subnet mask by starting (from the left) with NBits ones and HBits zeros afterwards.

For example, if NBits=20 and HBits=12

1111 1111.1111 1111.1111 0000.0000 0000

255.255.240.0

Repeat this step for each subnet.

Step 3: Filling-up the Table

Now that we know the subnet mask, we will need to find the rest of the parameters (subnet address, broadcast address,…etc)

S/N No S/N Address S/N Mask Broadcast Address Range of Host Addresses
0
1
2
  1. Calculating the Subnet Address:

The first thing to do is to re-order subnets in according to their BlockSize in descending order. This is extremely important.

The first subnet address is usually the main network address. Next subnet address can be easily calculated by adding 1 to the network part of the previous subnet address. Remember to do the addition in binary, so that carry bits can be pushed to the next octet (8-bits) on the left.

For example:

The first subnet 192.168.0.0 and the subnet mask is 255.255.255.240. In binary:

Subnet Mask for Subnet 0 (255.255.255.240)
1111 1111 . 1111 1111 . 1111 1111 . 1111       0000
Network (and also the address of Subnet 0, 192.168.0.0) Host
1100 0000 . 1010 1000 . 0000 0000 . 0000 0000
Subnet Mask for Subnet 1 (255.255.255.248)
1111 1111 . 1111 1111 . 1111 1111 . 1111       1000
Subnet 1 (192.168.0.16) Host
1100 0000 . 1010 1000 . 0000 0000 . 0001  0 000
Subnet 2 (192.168.0.24) Host
1100 0000 . 1010 1000 . 0000 0000 . 0001   1 000
Subnet 3, 4, 5,….

 

You can see in the table that the subnet addresses are calculated by adding 1 to the network-part of the previous subnet address.

  1. Subnet mask for each subnet was earlier calculated in Step 2.
  2. Broadcast address of each subnet is the last address before the next subnet. In follow-up of the previous example in 1, if subnet 0 address is 192.168.0.0 and subnet 1 is 192.168.0.16, then the broadcast address of subnet 0 is 192.168.0.15.
  3. Usable host range is simply all addresses between the subnet address and the broadcast address. Again following-up on the previous example in 1, and 2, for subnet 0 the subnet address is 192.168.0.0 and broadcast address 192.168.0.15, then the usable host range is 192.168.0.1-192.168.0.14

 

 

Example 1:

You have been assigned the duty of subnetting a network for an organization’s branch office. The headquarter’s IT manager said that you need to use the network address 192.168.12.0 with the default subnet mask (255.255.255.0). The number of hosts is shown in the diagram below.

Example 1 VLSM

Solution:

Since the main network address is already selected by the main office, we will directly start calculating the block size

HiHost=12

BlockSize= 12+2 rounded-up to the nearest power of 2.

BlockSize=16

Now we calculate the resulting subnet mask:

2HBits=16  ® 2HBits=24 ® HBits=4

NBits=32-4=28

Thus,

Subnet Mask is 1111 1111.1111 1111.1111 1111.1111 0000 ® 255.255.255.240

Now we start calculating subnets

Subnet Mask (255.255.255.240)
1111 1111 . 1111 1111 . 1111 1111 . 1111       0000
Network (and also the address of subnet 0, 192.168.12.0) Host
1100 0000 . 1010 1000 . 0000 1100 . 0000 0000
Subnet 1 (192.168.12.16)
1100 0000 . 1010 1000 . 0000 1100 . 0001 0000
Subnet 2 (192.168.12.32)
1100 0000 . 1010 1000 . 0000 1100 . 0010 0000
Subnet 3 (192.168.12.48)
1100 0000 . 1010 1000 . 0000 1100 . 0011 0000
Subnet 4 (192.168.12.64)
1100 0000 . 1010 1000 . 0000 1100 . 0100 0000

 

You can clearly notice the gradual increment in the subnet address within its first part on the right while host bits remain zero all the time.

Now we start filling the table:

S/N No S/N Address S/N Mask Broadcast Address Range of Host Addresses
0 192.168.12.0 255.255.255.240 192.168.12.15 192.168.12.1 – 192.168.12.14
1 192.168.12.16 255.255.255.240 192.168.12.31 192.168.12.17 – 192.168.12.30
2 192.168.12.32 255.255.255.240 192.168.12.47 192.168.12.33 – 192.168.12.46
3 192.168.12.48 255.255.255.240 192.168.12.63 192.168.12.49 – 192.168.12.62
4 192.168.12.64 255.255.255.240 192.168.12.79 192.168.12.65 – 192.168.12.78

 

First we start inserting the Subnet addresses, and the subnet mask. Then we start using the last address in each subnet as the broadcast address. Last column to fill would be the range of host addresses which will start from the first address after the subnet address to the last address before the broadcast address.

Example 2:

You have been assigned the duty of subnetting a network for an organization. The choice of class and network address is up to you. Take into consideration the host numbers mentioned in the diagram below.

Example 2 VLSM

Solution:

In order to make a proper network address and class choice, let us calculate the BlockSize first.

HiHost=541 ® BlockSize=541+2 rounded-up to the nearest power of two= 1024

Since the BlockSize is higher than 28 and lower than 216, we will choose class B

Let the network address be 172.16.0.0 and the default subnet mask be 255.255.0.0

Now we will calculate the subnet mask after subnetting,

BlockSize=2HBits ® 1024=2HBits ® 210=2HBits ® HBits=10

NBits=32-10=22

The chosen subnetmask will be:

1111 1111.1111 1111.1111 1100.0000 0000 ® 255.255.252.0

Now we start calculating subnets

Subnet Mask (255.255.252.0)
1111 1111 . 1111 1111 . 1111     11 00 . 0000       0000
Network (and also the address of subnet 0, 172.16.0.0) Host
1010 1100 . 0001 0000 . 0000     00 00 . 0000 0000
Subnet 1 (172.16.4.0)
1010 1100 . 0001 0000 . 0000     01 00 . 0000 0000
Subnet 2 (172.16.8.0)
1010 1100 . 0001 0000 . 0000    10 00 . 0000 0000
Subnet 3 (172.16.12.0)
1010 1100 . 0001 0000 . 0000    11 00 . 0000 0000
Subnet 4 (172.16.16.0)
1010 1100 . 0001 0000 0001    00 00 . 0000 0000

 

You can clearly notice the gradual increment in the subnet address within its first part on the right while host bits remain zero all the time.

Now we start filling the table:

S/N No S/N Address S/N Mask Broadcast Address Range of Host Addresses
0 172.16.0.0 255.255.252.0 172.16.3.255 172.16.0.1 – 172.16.3.254
1 172.16.4.0 255.255.252.0 172.16.7.255 172.16.4.1 – 172.16.7.254
2 172.16.8.0 255.255.252.0 172.16.11.255 172.16.8.1 – 172.16.11.254
3 172.16.12.0 255.255.252.0 172.16.15.255 172.16.12.1 – 172.16.15.254
4 172.16.16.0 255.255.252.0 172.16.19.255 172.16.16.1 – 172.16.19.254

 

First we start inserting the Subnet addresses, and the subnet mask. Then we start using the last address in each subnet as the broadcast address. Last column to fill would be the range of host addresses which will start from the first address after the subnet address to the last address before the broadcast address.

Unsolved Examples:

  1. You have been assigned the duty of subnetting a network for an organization. The choice of class and network address is up to you. Take into consideration the host numbers mentioned in the diagram below.

Example 3 VLSM

  1. You have been assigned the duty of subnetting a network for an organization’s branch office. The headquarter’s IT manager said that you need to use the network address 172.18.0.0 with the default subnet mask (255.255.0.0). The number of hosts is shown in the diagram below.

Example 4 VLSM

  1. You have been assigned the duty of subnetting a network for an organization. The choice of class and network address is up to you. Take into consideration the host numbers mentioned in the diagram below.

Example 5 VLSM