Author Topic: Oct Real World Whole Product results - Avast is last!  (Read 29438 times)

0 Members and 1 Guest are viewing this topic.

dagrev

  • Guest
Oct Real World Whole Product results - Avast is last!
« on: November 19, 2011, 02:55:53 AM »
I know some don't like these tests (which is fine), then this post is not for you, but those who are slightly interested in them.  As I've said many times before, they are not the end all, but they do say something.  As I and several others have pointed out avast is going backwards in detection and Oct placed them last. 

I'm just not sure how one can spin that into a positive, even if the test is off by some degree.  I'd like someone from avast to address this and explain why this might be and what is being done about it (not simply people who hate such tests, which serves no real point).

Make sure you sort by detection not vendor.
http://chart.av-comparatives.org/chart2.php

Gargamel360

  • Guest
Re: Oct Real World Whole Product results - Avast is last!
« Reply #1 on: November 19, 2011, 06:00:15 AM »
I'm one of those people who doesn't put too much stock in tests. ;D ;)

But this is still disheartening to see.  At the very least, it is bad publicity for Avast!.....As far as "spinning it to a positive", only a politician would be capable of such a feat.  Last place is last place, no matter how you spin it, this isn't the Special Olympics, everyone is not a winner.

That said, I'm not crying Hannibal is at the Gates....yet.  Mainly because we are speaking of single-digit percentage points of difference.  Last place by a nose, not a full length.  

But yeah....lets hope this backslide gets turned around...soon.

Offline RejZoR

  • Polymorphic Sheep
  • Serious Graphoman
  • *****
  • Posts: 9408
  • We are supersheep, resistance is futile!
    • RejZoR's Flock of Sheep
Re: Oct Real World Whole Product results - Avast is last!
« Reply #2 on: November 19, 2011, 08:08:06 AM »
I really hope avast! 7 will fill the gap...
Visit my webpage Angry Sheep Blog

Relu

  • Guest
Re: Oct Real World Whole Product results - Avast is last!
« Reply #3 on: November 19, 2011, 09:49:45 AM »
Hello,
I hope that avast team will take notice and improve the detection rate. This must be a priority.
Regards

warlock

  • Guest
Re: Oct Real World Whole Product results - Avast is last!
« Reply #4 on: November 19, 2011, 05:34:45 PM »
Whats the most worrying in this test is the steady decline tendency in detection rates when going through the results month by month.

Nesivos

  • Guest
Re: Oct Real World Whole Product results - Avast is last!
« Reply #5 on: November 19, 2011, 06:04:18 PM »
I know some don't like these tests (which is fine), then this post is not for you, but those who are slightly interested in them.  As I've said many times before, they are not the end all, but they do say something.  As I and several others have pointed out avast is going backwards in detection and Oct placed them last. 

I'm just not sure how one can spin that into a positive, even if the test is off by some degree.  I'd like someone from avast to address this and explain why this might be and what is being done about it (not simply people who hate such tests, which serves no real point).

Make sure you sort by detection not vendor.
http://chart.av-comparatives.org/chart2.php


Seriously:

You give us the backup on the tests and show us how the tests were done.

What other software was used?

The Muckety for the tester

How the test was constructed etc

The results of most tests can be significantly manipulated.

I am not saying these are.   But avast! now has over 170 million registered users.  How many does that website have?

DBone

  • Guest
Re: Oct Real World Whole Product results - Avast is last!
« Reply #6 on: November 19, 2011, 07:36:04 PM »
Results are results and making excuses about how the test was, or wasn't done, won't help the product...... Getting better will help the product.

Offline Para-Noid

  • Avast Evangelist
  • Starting Graphoman
  • ***
  • Posts: 6699
  • Trust only what you test yourself!
Re: Oct Real World Whole Product results - Avast is last!
« Reply #7 on: November 19, 2011, 07:38:22 PM »
I think I will wait for the "Whole Product Dynamic" (real world) tests.  ;D
But the decline in detection rate is disheartening.  ???

@ Nesivos av-comparitives. org has a strong history of tight methodology.  :)
Dell Inspiron, Win10x64--HP Envy Win10x64--Both systems Avast Free v17.9.2322, Comodo Firewall v8.2 w/D+, MalwareBytes v3.0, OpenDNS, Super Anti-Spyware, Spyware Blaster, MCShield, Unchecky, Vivaldi Browser and, various browser security tools.

"Look before you leap!" Use online scanners before you click on any link.

warlock

  • Guest
Re: Oct Real World Whole Product results - Avast is last!
« Reply #8 on: November 19, 2011, 07:40:37 PM »
@Nesivos
Learn something about av-comparatives, then you won't be making stupid remarks.

Gargamel360

  • Guest
Re: Oct Real World Whole Product results - Avast is last!
« Reply #9 on: November 19, 2011, 08:14:45 PM »
@Nesivos Learn something about av-comparatives, then you won't be making stupid remarks.
Wait, you understood what he posted?  ;D ;)
You give us the backup on the tests and show us how the tests were done.  What other software was used?  The Muckety for the tester
At least with a foreign language, I could run it through Google's translator......that is written in English, but I still have no idea what is trying to be said. ;D

Offline Vlk

  • Avast CEO
  • Serious Graphoman
  • *
  • Posts: 11655
  • Please don't send me IM's. Email only. Thx.
    • ALWIL Software
Re: Oct Real World Whole Product results - Avast is last!
« Reply #10 on: November 19, 2011, 09:14:54 PM »
I'd like to take this opportunity and comment little bit more about the test and the results.

First, let me say that the recent results (especially the Oct 2011 results) don't make me too happy. You're right that avast did quite poorly, and this needs to be fixed.

However, I have to object the accuracy of the name of this test (or other "Real-World" tests I have seen recently, for that matter). For starters, the test is done as follows: every day (or every few days), the tester pastes a few links to the browser's address bar. The links point to executable files that are downloaded from the Internet and then executed on the computer (unless the tested AV stops them). Thus, all modules, includes the Network and Web Shields are exercised, which makes the test much more "real-world" than the traditional on-demand tests. So far so good. That said, there's one principal problem here, which is - in the real-world scenario, how does the user (and/or the browser itself) get to these URLs? That is, why would the user (in a real-world scenario) paste such a link to the browser? What happens in the real-world is that the browser is typically redirected to such a URL through an exploit - most likely a Javascript exploit pack or something similar. And that's the part of the equation that the test completely ignores. It has been shown many times that one of Avast's main strengths is in its detection capabilities of such malicious scripts/redirectors which are generally working as the entry points for the actual malware (the payload). Our primary emphasis lately has been blocking of those entry points instead of focusing so much on the actual binaries which would then be downloaded to the computer if the exploit got executed.

Another thing that is worth discussing is the size of the test set. If you look at the chart, it may look terrible, but what it means in absolutely numbers is that avast missed 18 samples. Eset missed 11. AVG missed 10. Avira missed 8. Are the differences statistically significant? They probably are, but it certainly doesn't look like such a disaster when you look at the numbers like this.


In any case, we do understand the importance of doing well in these tests (and we do even have a limited understanding of the necessity of protecting our users better against those manually downloaded malware binaries) and that's why we're making some important changes in our upcoming avast version 7, due in Q1 next year. It will be very interesting to see whether the new version will live up to its expectations. Fingers crossed.


Thanks
Vlk
If at first you don't succeed, then skydiving's not for you.

Gargamel360

  • Guest
Re: Oct Real World Whole Product results - Avast is last!
« Reply #11 on: November 19, 2011, 09:37:35 PM »
I for one was aware of these issues, the tests not accurately testing the strong points of Avast! and the relatively low sample set.

Another point might be some of the highest scorers on that test are system gluttons.

Perhaps since the tests do not accurately represent your product, consideration might be given to withdrawing from the test altogether?  Although I suppose that could come with its own separate set of drawbacks. :-\   

Anyway, thanks for clarifying.  Please continue to think outside the box and ahead of the curve, and not let yourself be trapped trying to please testing methodology.

honestly....I'd be willing to bet the vast majority of your users have never even heard of AV comparatives

Offline Asyn

  • Avast √úberevangelist
  • Certainly Bot
  • *****
  • Posts: 76029
    • >>>  Avast Forum - Deutschsprachiger Bereich  <<<
Re: Oct Real World Whole Product results - Avast is last!
« Reply #12 on: November 19, 2011, 09:40:45 PM »
1. First, let me say that the recent results (especially the Oct 2011 results) don't make me too happy. You're right that avast did quite poorly, and this needs to be fixed.

2. Our primary emphasis lately has been blocking of those entry points instead of focusing so much on the actual binaries which would then be downloaded to the computer if the exploit got executed.

3. Are the differences statistically significant? They probably are, but it certainly doesn't look like such a disaster when you look at the numbers like this.

4. In any case, we do understand the importance of doing well in these tests (and we do even have a limited understanding of the necessity of protecting our users better against those manually downloaded malware binaries) and that's why we're making some important changes in our upcoming avast version 7, due in Q1 next year. It will be very interesting to see whether the new version will live up to its expectations. Fingers crossed.

1. +1
2. Well, this is a nice approach, but if such a binary should make it to the computer somehow (there are other ways than the net), it won't be detected.
3. Well, you know it, I know it and some others here around know it as well. The problem is, that many don't know it and also don't read the forum and especially they won't read your post here. So the majority of avast! users will (most probably) rely on these tests.
4. Yes, fingers crossed.
W8.1 [x64] - Avast Free AV 23.3.8047.BC [UI.757] - Firefox ESR 102.9 [NS/uBO/PB] - Thunderbird 102.9.1
Avast-Tools: Secure Browser 109.0 - Cleanup 23.1 - SecureLine 5.18 - DriverUpdater 23.1 - CCleaner 6.01
Avast Wissenswertes (Downloads, Anleitungen & Infos): https://forum.avast.com/index.php?topic=60523.0

Offline Rednose

  • Pirate Party Member
  • Avast √úberevangelist
  • Massive Poster
  • *****
  • Posts: 3742
  • Bits of Freedom : https://www.bof.nl
    • Nederlandstalig Avast! forum
Re: Oct Real World Whole Product results - Avast is last!
« Reply #13 on: November 19, 2011, 10:09:15 PM »
Thnx Ondrej  :)

Interesting explanation :)

Greetz, Red.
OS: Win 10 / iOS 17 / Debian 12 / Tails 6
Real Time: Avast Premium Security
On Demand: Malwarebytes
VPN: NordVPN ( NordLynx ) with Threat Protection ( Lite )

Nesivos

  • Guest
Re: Oct Real World Whole Product results - Avast is last!
« Reply #14 on: November 19, 2011, 10:09:51 PM »
Results are results and making excuses about how the test was, or wasn't done, won't help the product...... Getting better will help the product.

Maybe the test needs to get better.

Anything can be skewed to show false results.

Personally I think the test is BS

I run Malwarebytes regularly and it almost never finds anything that avast! misses.  Maybe a two files in the last two years or so.

I also run other AV programs in the on-demand mode and they haven't found much more than Malwarebytes has found.

That is plenty good enough for me :) :)
« Last Edit: November 19, 2011, 10:12:04 PM by Nesivos »