1.Introduction

The purpose of this document, it was to execute several efficiencies and detection tests in our endpoint solution, provided by Sophos, this document brings the result of the defensive security analysis with an offensive mindset performing a download manually and executing daily batches of malware sample created by MalwareBazaar in our environment.

Regarding the test performed, the first objective was to simulate targeted attacks using known malware to obtain a panoramic view of the resilience presented by the solution, with regard to the efficiency in its detection by signatures, NGAV, and Machine Learning, downloading these artifacts directly on the victim’s machine in manually way from daily batches provide by MalwareBazaar. The second objective consisted of analyzing the detection of that same malware (or those not detected yet) when they were changed directories, the idea here is to work with manipulation of samples (without execution).

With the final product, the front responsible for the product will have an instrument capable of guiding a process of mitigation and/or correction, as well as optimized improvement, based on the criticality of risks.

2.Scope

The efficiency and detection analysis had as target the Cybereason Endpoint Protection application (https://cloud.sophos.com) in Version :

• Agent Version = 10.8.9 VE3.79.0
• Core Agent – 2.10.7 BETA
• Endpoint Advanced 10.8.9.1 BETA
• Sophos Intercept X 2.0.17 BETA
• Device Encryption 2.0.82

Installed in the windows machine Windows 10 Pro;

Hostname – Threat-Hunting-Win10-POC, as you can see in the picture below:

3. Project Summary

The execution of the security analysis tests of the Threat Hunting team it was carried out through the execution of daily batches of malware sample created by MalwareBazaar with more than 1150 Malwares in a virtualized environment in a controlled way, simulating a real environment, together with their respective best practices of the security policies applied, the test occurred during 2 days, without count the weekend, along with the making of this document. The intrusion test started on the 24th of September of the year 2020 and it was completed on the 28th of September of the same year.

4. Running the Tests

3.1 Description

A virtual machine with a Windows 10 operating system was deployed to perform the appropriate tests, as well as the creation of a security policy on the management platform (Threat-Hunting–Win10-POC) e and applied to the due device.

The policy created was named Threat-Hunting–Win10-POC, following the best practices recommended by the manufacturer, and, for testing purposes, all due actions were based on an aggressive detection method.

3.2 First Test

The first stage of the tests was through the download of more than 1150 Malwares in a virtualized environment, that it was uploaded from a public repository known and maintained by the security community called MalwareBazzar (https://bazaar.abuse.ch/);

MalwareBazaar is a project from abuse.ch with the goal of sharing malware samples with the infosec community, AV vendors and threat intelligence providers.

MalwareBazaar creates daily batches of malware sample). The daily batches are created once a day at midnight (00:00 UTC).  Please consider that it takes a few minutes to create the batch. So, I kindly ask you to not fetch the daily batch before 00:15 UTC.

The day was chosen for this test it was 2020-09-15

(https://mb-api.abuse.ch/downloads/2020-09-15.zip);

The purpose of this test was to simulate the same process as a user receiving a zipped file (.zip) and performing the extraction of these artifacts in their own environment.

During this test, one thing called my attention:

• The first Detection happened on September 24, 2020, at 11:14 AM GMT-3
• The last Detection happened on September 28, 2020, at 12:11 AM GMT-3

That is, we have a time gap with more than 4 days between the first and the last detection that was the time it took for malwares to be detected

After performing the action of extracting the files, it was possible to verify that Sophos Security Endpoint starts the blocking process, following the best practices a function named Outbreak Detected is called, this feature enables when any device has experienced 100 detections in a minus 24-hour period. This is to avoid flooding an admin with similar or repeated detection events during a malware outbreak.

During an outbreak the priority should be dealing with the detections on the device, so you may want to take this machine off your network while it is being investigated.

The important point here is:

We had more than 1150 malware, and after the outbreak, detection to be started, we can see just 100 malware blocked, even though our virtual machine had more than 1000 we can see that the behavior detection process happens this way, first Sophos endpoint detect after that clean, but for us, this process it was very slow.

If you are a Sophos Intercept X customer and would like more information on what led up to the detection, use the Threat Cases feature in the Sophos Central Admin console to see what was happening on the computer at the time. (Information provide by Sophos Support)

Source: https://support.sophos.com/support/s/article/KB-000037174?language=en_US

If you are unsure how to deal with the outbreak please contact Sophos Support

If you believe the detections that led to the malware outbreak event being reported are incorrect (false positive) please see article: How to investigate and resolve a potential False Positive / Incorrect Detection

Once you have resolved the cause of the outbreak and the threat (or PUA) has been removed, you need to mark the Outbreak event as resolved in the Sophos Central Console.

So, after to mark the machine as solved, we following wait for the detection process, and for our surprise, just 232 malwares blocked as you can see below

3.3 Second Test        

The second stage of the tests was through the transfer of folders to another directory within the same machine, the purpose of this test was to simulate a transfer of files within the same environment.

When a new file is generated on the disk, soon we should have a new entry in a block of that disk, and in theory, the antivirus should take some action (considering that it has the real-time enabled), we could define it as a file manipulation (still not running) where the endpoint protection is already necessary, considering that a new directory was created, soon we would have a new repository with several hashes inside to be examined.

During this stage a very strange situation caught our attention, only 86 malwares were moved to a new folder, many information’s about the Detection process and/or Detection flow aren’t clear how works, after that we saw many alerts about possible blocks and detections, but we didn’t get to find any information clear about blocks.

The victim machine stayed with many alerts during this test, and CPU and Memory had a very high consumption in the machine, during this test, from our perspective looks like the detection engines got lost in all detection process, after that we need to execute the reboot of this machine, to try to understand with the alerts will be finished or may some malware will be detected.

Our expectation was that there would be many detections, but after almost one hour there was no new detection by Sophos Endpoint Protection, so, remained 787 malware from the first folder not detected and 86 malware from the second folder not detected as well.

3.4 Third Test  

The third stage of the tests was through the use of the ScanNow action by Cloud Sophos, to perform a complete scan on the machine, manually, in this way, all malware should be eliminated, as they are already known malware as mentioned earlier.

After performing this third test and the execution of the Scan Now feature, we saw the same behavior in our environment, the Windows 10 machine stayed with many alerts during this test, and CPU and Memory had a very high consumption in the machine, during this test, for our perspective looks like the detection engines got lost in all detection process, after that we need to execute the reboot of this.

We tried to execute another test not pretending in the beginning, as we have much malware not detected by Sophos Endpoint Security, we ran a Scan provide by Agent installed in Windows 10 Machine.

However, at the end of this test, unfortunately, it was not clear something about the product, like a detection time, flow of detection engines, machine performance, and mainly inefficient of detection, when we open the Sophos Central today (28/09/2020) in the Windows 10 Machine, we found that the console was very alert with detections but that depended on an individual action by the support team, this team needs to apply “Marked as Resolved”

Before perform the task (Mark as Resolved), we access the Windows 10 Machine again and we see another time many alert pop-ups in the user machine as you can see below.

So, to finalize this test we deleted all files manually, because the Sophos Endpoint Security did not detect more than 600 malware, and we performed the task of the MARK AS RESOLVED.

5.  Impact and Risk

 

At the end of this test, it was possible to verify that this solution can bring many risks to our environment, even when it comes to known malware, whether executed inside any environment, may perform an infection.

• After the first extraction, just a few samples were detected;

When it comes to a major malware infection we can have several types of attack vectors, so it is very important we have efficient detection.

• Endpoint went into Outbreak Prevent mode;

A very interesting feature, but the risk is, we can stop a user or machine, just because received much malware in 24 hours, it doesn’t mean that is an outbreak, like a pandemic, and it’s not clear, how the engines work after this break.

• High CPU and memory consumption during the detection process;

Very high load on CPU and Memory used by many processes requested by AV, which directly impacts the user experience.

• Necessity to reboot the machines more than once, due to alerts and Outbreak Prevent mode;

Due to the high load on CPU and Memory, it was impossible for the machine to continue to work well, there was a necessity for the machine to reboot.

• After second test more than 80 Malwares not detected when moved to another folder;

Due to the high load on CPU and Memory, high alert pop-ups, and maybe the outbreak prevent feature, It was not clear how the engines would work when any sample has been written in a new block part of the disk.

• After the ScanNow and the end of this teste, we havbe more than 600 Malwares not Detection, that is, just 54% in success detection

More than 600 Malwares inside on Windows 10 machine, note detected by Sophos Endpoint Solution, totally dangerous, if you see in the Sophos Central, you’ll find almost 600 Detection logs “Events reports”, that is, we had 1159 malware, many there were not detected that whether executed could take a big infection on this machine.

6.  Recommendation Actions

As we mentioned before, the idea it was executed the test in many malware, and this case, for this reason, to be totally known the following actions will be taken to improve the protection environment of our assets:

• This report should be sent to Sophos Security Team to validate with them how the detection flow for known malware works, and why all those malware didn’t were detected;
• Validate the performance of NGAV, Machine Learning, and other components, regarding this type of detection;
• Validate why we have high consumption in the machine with CPU and Memory during the detection process.
• Validate why we have high consumption in the machine with CPU and Memory during ScanNow and Scan provide by a local agent.
• The best practices of the configurations will be revalidated with the Sophos team;

7.    Answers from Sophos Company

As we mentioned before, the idea has executed the test in many malware, and this case to bring the result of the defensive security analysis with an offensive mindset a download manually and executing of daily batches of malware sample created by Malware Bazaar in our test environment.

We sent this email request information’s to the Sophos Support team on September 28th as you can see below

I sent another email on October 06th and another email on October 09th, all of that with Reports showing many tests that we did in Sophos Endpoint Solution.

After almost 15 days, I received I generic answer from the Vendor as you can see below:

“We’ve similar things like this before.  To be honest, this is an unrealistic test.  It does not represent what happens in the real world.  An attacker is unlikely to drop that many new and unique pieces of malware – they would likely cause their attack to become unstable. When reputable security test operations conduct testing, they deliver the

threats in sequence, just like what would happen in the real world.

 Looking at how this report was done, I can think back to when Cylance and Deep Instinct run tests like this with malware from Virus Total, Malware Bazaar, and another corpus of malicious code

Direct them to places that test anti-malware tools and how they’ve devised the right methods when testing.

 NSS Labs: https://www.nsslabs.com/reports/endpoint-security-test-methodology-v5-0/

SELabs: https://selabs.uk/wp-content/uploads/2020/04/04-predictive-malware-response-testing-methodology.pdf

 The method of ‘dumping a bunch of PEs onto disk’ is NOT representative of the real world and in doing so one should expect system resources to peak whilst the real-time protection attempts to deal with the massive influx of malicious entities.  Simple facts, nothing surprising here.”

Name: Filipi Pires

Principal Security Engineer and Researcher at Zup Innovation

São Paulo – Brazil