Reputation Based Detection with Python

The purpose of the reputation based detection systems is detecting low reputation behaviors (Low reputation file opening, requesting IP address). The suspicious situation will be noticed when requesting low reputation IP address within a network using this system.

I will show basically how to create reputation based detection system. I'll use 3rd party resources for reputation data

Some data sources:

malwaredomainlist.com

SANS

abuse.ch

‍

The system basically consists of 3 stages:

1. Monitoring network traffic
2. Collection data from 3rd party sources for reputation data
3. Comparison of addresses that devices communicate with and data from sources

With Letsdefend.io, you can review reputation-based cases in a real SOC environment.

‍

Monitoring Network Traffic

For investigating data, clients has to forward network logs to security server. In this blog, logs are recorded with "tcpdump" and directed in ".pcap" format.

‍

Collection data from 3rd party sources for reputation data

Some of the IP addresses associated with malwares in the past can be obtain from the list of shared by “abuse.sh”. The data is written to “IP_list.txt” file with the simple script like in the image below.

“IP_list.txt” content:

In order to add new data to "IP_list.txt" file, we must be collecting data from 3rd party service daily. With the help of "Crontab" on the Linux systems, "get_IPs.sh" script could run everyday automatically.

I use the command “crontab -e” to define new job in crontab and add  "0 18 * * * /home/ogunal/get_IPs.sh" to the bottom line to make my script run at 6pm every day.

At this stage. we ensured that the data from the 3rd party source is renewed daily.

3-Comparison of addresses that devices communicate with and data from sources

In this stage, we extract IP addresses from logs and compare 3rd party sources data. If a match occurs, we will generate an alert.

First, i prepared this function for extracting IP addresses from logs.

At stage 2nd, prepared another function that reading IP_list.txt file and forwarding data to an array.

Taking the intersection of the 2 existing lists, I prepared the code that determines whether there is any communication with the IP addresses with low reputation.

When i run the application, if there is any of the IP addresses listed in the log records, it is generating an alert.

Problems of the reputation-based detection system

• If IP address depends on more than one domain:
• On the one hand, there is a domain used for harmful purposes, on the other, an ordinary blog can be found.
• In this case, requests made to the harmless blog will also appear suspicious by the system due to its IP address.
• Change of owner of IP address
• If owner of the IP address changed and IP still one the blacklist. Requests to new content will also be considered suspect.