Combinatorial AV testing

[Bob Anderson]

This site

http://winnow.oitc.com/AntiVirusComboQuery.php

has a very strange 'combinatorial' analysis of AV's including Avast!

I have some very strong comments to make, but I'd like others to take a look first.

-Bob

[DavidR]

My first thoughts were this was some sort of comparison rather than using them in combination.

I and many others don't believe you should run two resident anti-virus scanners together on the same system as there is a likelihood they will conflict and far from giving additional protection could offer less.

Though some of the AV options on the site are capable of being run as on-demand scanners the majority are resident scanners.

I humoured the site by entering 4 AVs, avast, avg, antivir and kaspersky and the level of protection was given as 84% which is pathetic as all of the individual AVs get better than this in av-comparatives.org tests. In fact most are in the 90% plus category.

So not very impressed.

[psw]

It is statistics for early (near 0-day) infection. The authors describe the criteria of inclusion:

Periodically, our system checks for MIRT updates. If there are some, we evaluate the aggregate performance of all the antivirus engines. If the aggregate performance indicates the malware is still at an early outbreak level and that only a few (<~25%) of the reporting antivirus systems are detecting the each submitted malware, the information on the performance of the antivirus engines is entered into our databases.
The statistical results displayed are representative of the ability of each antivirus system's ability to deal with early (near 0-day) infection outbreaks. The performance by each antivirus system on early detections does not reflect the overall performance of an antivirus system for general system cleaning. Further, our testing methods specifically do not assess false positive rates.

So this comparison reflects mainly the speed of antivitus bases updating.
As for combinatoracal values it is not clear the precise algorithm of calculation. Namely, are the correlation in detection between different AV taken into account or not?

[DavidR]

I didn't stick around to read anything on the site based on the fact I couldn't see the purpose of this as it is advised not to have multiple resident AVs installed on the system. Not to mention the word 'Combinatorial' which to a mere oik went right over my head

Had to go looking, http://www.tfd.com/combinatorial and http://en.wikipedia.org/wiki/Combinatorics.

[Bob Anderson]

Well, I think we can all agree that the first problem with the site I linked to is that by implication it leaves the impression that it is all right to run multiple resident AV's at the same time. This is of course just asking for trouble. I posted this link to a number of sites, and sure enough some replies took this to mean it was all right.

The so-called combinatorial analysis makes no logical sense to me. I suppose if one ran on-line web (not resident) AV scans it might make some sense. For example, I punched in Avast!, AVG, Kaspersky and Symantec and it came up with a 75% detection rate of near-zero day malware. I presume that means that if I was to run these four AV's sequentially on-line via the web, it would detect 75% of near-zero day malware. There is merit in the idea that once a resident AV finds an infected file, then sometimes running an on-line scan from another maufacturer can help in determining its real or false positive status.

But what I find unforgivable about the site is that it (unintentionally, no doubt) implies that it is OK to run multiple resident AV's and does not explicitly state that this is a very bad idea.

-Bob

[DavidR]

If you are looking for confirmation, good or bad detection, I think VirusTotal is the best bet for single file analysis as it has 32 scanners.

You can check the offending/suspect file at: VirusTotal - Multi engine on-line virus scanner.

[psw]

The problem is what kind of algorithm is used for combined detection rate calculation. For example, suppose that we scan using AV with detection rate 70%. So with probability of 30% virus will be undetected. Now we add the 2nd AV with detection rate 60%. But we can not suppose that combined probability of undetection will be (1 - 0.7) * (1 - 0.6) = 0.12 due to the correlations between different AV. In the extreme case all samples which can detect AV with lower datection rate can be contained in already detected samples by AV with higher detection rate. In this case addition of the 2nd AV gives nothing.

[Lisandro]

In the extreme case all samples which can detect AV with lower datection rate can be contained in already detected samples by AV with higher detection rate.

Generally this is not the case, you will agree...
The better detection rates are with the antivirus with more detection capacity and it's not just for luck that they detect more. Generally, faster is better in antimalware world due to in the wild detections.

[dw2108]

The referenced comparison above is combinatorial only ostensibly. In reality, it is a very unusual exercise in stochastic automata complexity theory, and appears ad hoc as such without further remark.

Dave -- The idiot who placed two different operating systems on the SAME partition!

[AssistantX]

"If used in combination, there is a 23 percent probability of detecting early malware outbreaks using Avast, Avast, Avast and Avast in combination."

 

Seems to use sites like VirusTotal to test if a group of "zero-day" malware are detected and uses those percentages.