All tests on The PC Security Channel are easy to understand and completely transparent. Every test is on video and you can use the data under results to make your own judgements. Before you do that though, it is important to understand the test process and what each figure represents.
All products are tested with default settings, with PUP/PUA detection enabled, and are set to delete or quarantine any threats detected.
Each test is conducted in two phases based on the type of malware samples used. Keep in mind this is more of guideline and not a hard distinction as every individual sample isn't tested or clustered.
PHASE 1 This is the preliminary phase and includes fewer samples (typically <200) mostly consisting of ransomware, trojans and other malware with a low concentration of PUPs.
Phase 1 is considered a baseline test with a pass/fail criteria. If the test product does not maintain at least a partial clean sheet in this part of the test, it is disqualified.
PHASE 2 A much higher number of samples are used in this phase with a moderate concentration of PUPs and adware/grayware.
CLEAN SHEET RECORD
This is the most comprehensive and concrete indicator of product performance and we suggest that you give it the highest priority. As this metric is solely based on infections found on the test system, it is immune to any duplication errors, false positives in the samples used etc. Below is a legend of each possible result and what it means.
YES No infections were found on the test system ie. the product successfully blocked all malware from causing any permanent damage
(This may exclude minor PUPs and grayware).
PARTIAL There is no absolute evidence of damage to the test system, but results indicate presence of malware (or major PUP infestation).
NO Test system was infected, malware persisted after reboot causing permanent damage.
All the three states above are represent the assessment at the end of a full test (after Phase 2).
FAIL The product failed to keep a relatively clean system during the initial baseline test (Phase 1) and was thus disqualified.
Pro-active detection: This is the primary percentage metric and indicates what percentage of the executed malware was successfully blocked by the test product, prior to execution. This does not include reactive detection and removal, which is why it is recommended that you give the clean sheet record highest priority.
AUTOMATED TEST PROCESS
Every test is automated with a python script (malex) that executes samples one by one on the test system with the test product up-to-date and real time protection enabled. Also, please note:
The malex script itself is not malicious, it is a harmless tool.
Each malware is executed as a separate process group, and a breakaway job. This ensures that the test continues smoothly regardless of the outcome and the malware process functions independently of the testing script.
For the pro-active test:
If a sample fails to execute, for whatever reason, the test-product is granted a block.
If a sample successfully makes it to memory and begins execution, it is considered a miss.
The removal rate is calculated based on:
The number of items in the samples folder at the start of the test
The number of items in the samples folder at the end of the test