File carving is a technique that’s been around a while and traditionally has uses in data recovery and forensics. The origin traces back to the idea that nothing deleted on a computer is truly gone, until or unless that memory has been written over or wiped.
Conventional definitions of file carving often refer to this as memory reallocation. What this means is even if you delete a file on your computer, file carving can be used to reconstruct that file, until that memory is reallocated to saving other data.
Technopedia puts it this way:
“Part of the success of file carving relies on the idea that files that are deleted from a computer or device are not really completely lost until their memory locations are deleted during a device wipe or other fundamental sweeping away of residual data. In many cases, file carving can be part of data forensics, where law-enforcement professionals or other specialized experts can reconstruct files, even after something like a disk formatting, or when the user has effectively deleted the files from a drive. Since many of the fragments of the file may still rest in unallocated memory, they can theoretically be reconstructed.”
File Carving in Network Security
What does file carving mean to network security?
Modern standalone intrusion detection systems (IDS) “carve” files in essentially the same way. The difference is that the IDS sensor monitors the connection between the client and server and uses the data from the higher-level file transfer protocol (like HTTP or FTP) to reconstruct the file.
It’s important to note that IDS isn’t blocking the traffic as an intrusion prevention system (IPS) might. Instead, the detection mode allows those files to continue to the receiver and forwards the reconstructed file to an appropriate engine for analysis.
If the carved files contain characteristics of malware, the file will be “convicted” as malware triggering security alerts in order to mitigate the threat. Since the malware conviction engine is embedded in the IDS sensor, the entire process happens in fractions of a second. Typically, this is referred to as network speed or “line speed.”
This speed is important to the business, as there is always a balance between detection efficacy and performance. Not only do incident responders want as much notice as possible regarding potentially hazardous payloads entering the environment, but only detection techniques that can occur in milliseconds can be considered for real-time blocking.
Also see these related posts:
White paper: Bro vs. Snort or Suricata
The Bricata Solution for Retail [data sheet]
The Risk of Overconfidence in the Cybersecurity Perimeter
The Sandbox and Probability of Failure
To understand the benefits of file carving in combination with a malware conviction engine, it’s useful to compare it to sandboxing. The comparison here is for the sake of illustration and to present use cases; it is not to say one technique is better than the other.
Sandboxing is a technique where a suspected file is copied from the network and placed in a test environment. The test environment strives to mirror a designated machine. For example, if the suspect file was traveling to a Windows client, the sandbox pretends to be that machine.
The purpose of this is to allow the file to execute in a controlled environment and see what happens. If in fact, the file is malicious, then it is already off the network. If it proves benign, it’s allowed to proceed back into the traffic pattern.
There are advantages and disadvantages to this technique. On the positive side, the system gets a detailed view as to exactly what the malware sample does as an attack unfolds.
There are two key challenges with this method on a high-speed network. First, it can take several minutes for the process to be completed. If there are many suspected files, they can start to queue waiting for a turn in the sandbox.
As an incident responder, having the additional information provided by the sandbox is a luxury, while knowing the file was malicious as quickly as possible is the highest priority. Detailed forensics can be run later, if necessary, and only after the environment has been secured and remediated against the threat.
The second challenge is, sometimes malware requires multiple parts to execute. For example, the program may be split into a separate dynamic link library (DLL) file. In this case, if a malware is routed to the sandbox, it won’t execute in that safe environment because it can’t communicate with the other part of the file.
In security circles, this is referred to as the probability of failure. The sandbox needs the whole file to make a determination. By contrast, the IDS sensor with a malware conviction engine does not.
How does Bricata do this?
Bricata uses a combination of tools to secure networks. In the case of file carving and the conviction engine, we use an open source tool called Bro IDS and an algorithm licensed from Cylance (note: the name “Bro” stems from “Big Brother” and not “bro culture”).
Bro is different from traditional IDS tools because it is focused on network analytics. Bricata uses Bro to monitor for network anomalies, but it also gives us the file carving capability.
Once the file is carved, the Cylance engine examines the payload files for the characteristics of malware. Because it’s looking just for characteristics associated with malware – it does not need the whole execution environment to make a conviction. As such, Bricata can identify malware not only in stand-alone executable files but also with DLL file objects that are typically problematic for sandbox environments to process.
Most importantly, this engine is able to render a verdict in near real-time, whereas a sandbox may take between 30 seconds and five minutes to render a verdict (or longer if a processing queue exists). As a lot of damage can be achieved by malware variants such as ransomware if a five-minute time period, this difference can be crucial for incident response.
It’s important to know it’s the combination of these two concepts – file carving and malware conviction – working together on the same sensor placed inside the network that provides this capability.
The Bricata solution includes three detection engines, not just two. More recently, we released a software upgrade with expanded capabilities for security alert triage and threat hunting.
If you enjoyed this post, you might also like:
The Race Against Time Between Vulnerability and Patch