Web Proxies generate a common set of information that can be used for threat hunting and detection. These information contains
Duration, HTTP Status, Bytes In, Bytes Out, Protocol, HTTP Method, HTTP Version, URL Category, URL Hostname, URL Path, URL Query, Mime Type, FIle Name, User Agent.
Below, I explained how we can use this information to hunt or detect threats.
Duration
This information shows how long the transaction has taken. Malware can communicate with the C2 server over the HTTP(S) protocol. When this is the case, it asks for commands periodically. This period doesn't have to be a constant value like every 10 minutes. Malware can also use jitter to make random-looking requests. Also, keeping the connection open can also be used by malware. In any case, it needs to either ask for commands very often or keep the connection open.
Technique
Calculate the sum per SourceIP-DestinationIP pair over 12/24 hours
What to look for
Higher values may indicate beaconing. Keep in mind that not all beacons are malicious. That's why we are hunting.
Note : If you apply the same method to your public websites, you can detect web scraping or customer data scraping.
HTTP Status
Users visit websites, post something, sometimes upload some data, or download a file. In normal conditions, these transactions have an HTTP 200 result. When it comes to malware, it is possible to use HTTP error codes as a C2 channel. Also, most malware use DGA(domain generation algorithm) in order to keep the connection persistent if one of the domain is blocked. In such a case, the malware keeps getting HTTP errors and tries the next domain.
Technique
1. Calculate the total count of the HTTP Status Codes per SourceIP or per SourceIP-DestinationIP over a specific time period.
2. List URLs having only HTTP Errors.
What to look for
1. Higher values of an uncommon HTTP Status Code may indicate C2 activity.
2. Higher values of HTTP errors for a website can indicate failed C2 activity.
Bytes In
In normal conditions, when a user visits a website, downloads a file, etc., each transaction has a different size. On the other hand, malware visits the same page every time. This makes the downloaded content has the same size unless the attacker starts interacting with the victim machine.
Technique
1. Calculate the count of BytesIn per Source-Destination pair over 12/24 hours. You have the best chance when the attackers sleep as there is no interaction.
2. Calculate the ratio of count(BytesIn) per Source-Destination pair. This is for comparing the attacker interaction versus idle status.
What to look for
1. Higher values may indicate beaconing. C2 servers reply with the same data, making Bytes In value the same.
2. Higher values of ratio may indicate C2 beaconing.
Bytes Out
A normal user activity consists mostly of downloading data. Uploaded data is usually small unless there is a file/data upload to a website.
Technique
1. Calculate the sum of BytesOut per Source-Destination pair over 12/24 hours.
2. Calculate the ratio of count(BytesOut) per Source-Destination pair over 12/24 hours.
What to look for
1. Higher values may indicate data exfiltration.
2. Higher values of ratio may indicate beaconing.
HTTP Method
In normal circumstances, a user web traffic contains a large amount of HTTP GET, a small amount of HTTP POST methods. Other HTTP methods, such as HTTP PUT, are expected to be seen less.
Technique
Calculate the ratio of the POST or PUT over GET per Source-Destination over 4/8/12/24 hours.
What to look for
Higher values of ratio may indicate beaconing or exfiltration.
URL Hostname
Usually, a user visits websites that are in the top 1M list. In some cases, an unpopular website can be visited by lots of users as well (think about 3rd parties having business with the company).
Technique
1. Compare with top 1M domains and calculate the hit count.
2. Calculate hit count per Hostname.
What to look for
1. Hit count <5 and Hostname is not in the top 1M may indicate malicious payload delivery.
2. Small number of hit count may indicate malicious payload delivery.
URL Path
When an attacker compromises a website and uses it as a C2 server, the malware most probably uses the same URL Path for C2 communication.
Technique
Calculate count per Source-Destination-URLPath pair.
What to look for
Higher values may indicate beaconing.
URL Query
URL query information is seen when you search for an item on a website. Malware does the same when asking the C2 server if there is anything to run on the victim machine. The query can be encoded/encrypted as well.
Technique
1. Calculate count per Source-Destination-URLQuery.
2. Calculate the length of URLQuery.
3. Look for base64 encoded strings in URLQuery.
What to look for
1. Higher values may indicate beaconing.
2. Higher values may indicate encoded data, a sign of exfiltration or beaconing.
3. Encoded strings may indicate beaconing or exfiltration.
Mime(Content) Type
Unfortunately, most web proxies fail to determine the exact type of content.
Technique
List mime type per Source-Destination pair.
What to look for
Uncommon mime types may indicate a malicious file.
User Agent
Normally, all applications have their own user agent information. Malware can try to mimic a legitimate application user agent but sometimes fail to do that with a small typo.
Technique
Calculate count within the environment(long tail analysis).
What to look for
Lower values may indicate a malicious binary existence.
URL Category
In most environments, there are commonly blocked web categories like Hacking, Pornography, Dynamic DNS, etc. Uncategorized web sites are a pain and sometimes this category has to be allowed for the sake of business continuity.
Technique
Query for Uncategorized, Dynamic DNS, and other suspicious categories. Calculate dcount of SourceAddress by URLHostname.
What to look for
Small dcount values may indicate abnormal/suspicious/malicious activity. If an uncategorized URL is visited by many users, it is less likely that the URL is malicious.
HTTP Version
There are four HTTP versions — HTTP/0.9, HTTP/1.0, HTTP/1.1, and HTTP/2.0. The current version is 1.1 and the future one will be 2.0.
Technique
Check HTTP versions
What to look for
HTTP/0.9 and HTTP/1.0 are old. This may be an indication of malicious activity.
Protocol
Web proxies are able to determine the protocol by analyzing the traffic.
Technique
Compare ports with their standard protocols.
What to look for
Common Protocol-Uncommon Port or Common Port-Uncommon Protocol may indicate malicious traffic.
File Name
It's not always possible to properly log names of the files that are downloaded from the internet. If it's logged properly, file names can be used for hunting. Some malware droppers download randomly named files.
Technique
Entropy analysis on filenames.
What to look for
May indicate malicious payload delivery.
Hope this guide will help you.
