Author Topic: The truth about av comparatives? How do they get paid?  (Read 22629 times)

0 Members and 1 Guest are viewing this topic.

Offline Lisandro

  • Avast team
  • Certainly Bot
  • *
  • Posts: 67275
The truth about av comparatives? How do they get paid?
« on: November 27, 2011, 08:12:23 PM »
Well, the discussion arise about how independent is av comparatives?
Do they get paid by the av companies?

Seems they're paid for testing the product?
They're paid to publish the results?
If the results are good ones, well, do you need to pay more?
They're paid to release/share the samples that were tested?

I'm publicly quoted in Melih's blog (Comodo CEO):
http://www.melih.com/2011/11/27/av-comparatives-org-bullying-censorship-and-financial-deals-with-anti-virus-vendors/

Melih said:
AV companies pay these testing organisations.

I've answered:
Not all of them.
I was clearly thinking on avast company (my antivirus).

He said:
so AV comparitives don't get money from vendors?

I've finally questioned (:
Are they lying?
http://www.av-comparatives.org/en/about-us

Or let us say clearly: are they results influenced by money or paid - in this sense - by the av companies? Aren't they independent in any way?
Of course, regardless they are a non profit organization, they need money to live...
I have an NGO also. I'm completely independent of my financial sources.

And then:
But, do they? Before the testing or after (to publish)? Are the results influenced by this?

Can avast team post what is the real financial relationship with av comparatives?
Is it independent?
Can we trust it?


I've posted both on Comodo forums and Wilders:
https://forums.comodo.com/melihs-corner-ceo-talkdiscussionsblog/avcomparativesorg-bullying-and-financial-deals-with-anti-virus-vendors-t78869.0.html;msg564749#msg564749
http://www.wilderssecurity.com/showthread.php?t=312965
« Last Edit: November 27, 2011, 08:16:54 PM by Tech »
The best things in life are free.

Offline !Donovan

  • Web Analyst
  • Avast Evangelist
  • Super Poster
  • ***
  • Posts: 2220
    • The WAR Against Malware
Re: The truth about av comparatives? How do they get paid?
« Reply #1 on: November 27, 2011, 08:17:44 PM »
Interesting Tech. :o

Could this be why McAfee scored at the top this time?
Familiarize Yourself! | Educate Yourself! | Beautify Yourself! | Scan Yourself!
"People who say it cannot be done should not interrupt those who are doing it."

Offline IBK

  • Jr. Member
  • **
  • Posts: 67
    • AV-Comparatives
Re: The truth about av comparatives? How do they get paid?
« Reply #2 on: November 27, 2011, 08:22:58 PM »
Hi,

as stated in the methodology document of 2008, vendors pay for the services.
For the public main test series, every vendor that applied for inclusion (if he gets included) pays the same fee, in order that we can cover the costs of the tests etc. Of course, it does not have any influence on the results. Comodo never applied for the public main test series, as in that case results are being published in any case.

Instead, Comodo paid us several times for internal/confidential single product detection rate tests. As they are confidential, we can not publish them if Comodo does not want to. Comodo decided to keep all tests they commissioned to remain internal. If they would have wanted to use them in marketing (i.e. use the logo etc.) and to have it published together with a review, they would have had to pay additionally.

There are many wrong perceived things written (in a misleading way) by Comodo, so we may reply soon to clarify at least some of them.

Offline Lisandro

  • Avast team
  • Certainly Bot
  • *
  • Posts: 67275
Re: The truth about av comparatives? How do they get paid?
« Reply #3 on: November 27, 2011, 08:28:37 PM »
Thanks for joying IBK.
Well, seems the financial relationship with av companies is now more clear.
But, in my opinion, companies should NOT pay to be reviewed in main tests. Otherwise, I can't really think on independence or full trustfulness. Well, the results won't be complete as some av products weren't tested as they haven't paid for it. Am I wrong?
The best things in life are free.

Offline bob3160

  • Avast Überevangelist
  • Probably Bot
  • *****
  • Posts: 43694
  • 60 Years of Happiness
    • bob3160 Protecting Yourself, Your Computer and, Your Identity
Re: The truth about av comparatives? How do they get paid?
« Reply #4 on: November 27, 2011, 08:35:13 PM »
Thanks for your reply IBK,
I've always been told that you are an independent lab and aren't paid by companies that make they product.
Your statement sort of shoots wholes into my prior belief.  :'(
How do I really know how well any AV scores if the results depend on you receiving payment ???
Free avast! Security Seminar: http://bit.ly/2N1eaR2  -  Important: http://www.organdonor.gov/ -- My Web Site: http://bob3160.strikingly.com/ - Win 10 Pro v1909 64bit, 24 Gig Ram, 1TB SSD, AvastOmni 20.3.xxx, How to Successfully Install Avast http://goo.gl/VLXdeRepair & Clean Install https://goo.gl/t7aJGq

Offline IBK

  • Jr. Member
  • **
  • Posts: 67
    • AV-Comparatives
Re: The truth about av comparatives? How do they get paid?
« Reply #5 on: November 27, 2011, 08:38:08 PM »
Due to the complexity of tests and the work behind it, there is no testing organization which could live from nothing or user donations (I tried that at the beginning). I think that test results should be made available for free to the public; users should not pay for the external QA etc. of AV vendors.
We have a waiting list of vendors which would like to get included in our tests, but we test only ~20 products in the public main tests. As all the ones which get included (if they qualify) pay the same fee (even if they took the given opt-out option for the retrospective test), I see no way how it could influence the results. It is a very insulting to insinuate that results are influenced in any way.
Furthermore, my goal is and always was to provide independent tests (that's also one reason why I opted for a NPO instead of a company focused on making profits), so I get quite angry if I see people trying to bring us in bad light for their own purposes.
« Last Edit: November 27, 2011, 08:41:13 PM by IBK »

Offline Lisandro

  • Avast team
  • Certainly Bot
  • *
  • Posts: 67275
Re: The truth about av comparatives? How do they get paid?
« Reply #6 on: November 27, 2011, 08:47:02 PM »
I can't see the reason for the opt-out methodology then.
The users are being conducted to a false sensation of completeness and security.
The results should be public and the av company should agree with this to join the test.
The best things in life are free.

Offline bob3160

  • Avast Überevangelist
  • Probably Bot
  • *****
  • Posts: 43694
  • 60 Years of Happiness
    • bob3160 Protecting Yourself, Your Computer and, Your Identity
Re: The truth about av comparatives? How do they get paid?
« Reply #7 on: November 27, 2011, 08:49:06 PM »
@IBK,
My intention was never to insult any one but simply to find out if my belief in the
impartiality of these tests was true or a myth.
Since I use some of these test results, I'd like to make sure that they are as independent as I
make them out to be.
Free avast! Security Seminar: http://bit.ly/2N1eaR2  -  Important: http://www.organdonor.gov/ -- My Web Site: http://bob3160.strikingly.com/ - Win 10 Pro v1909 64bit, 24 Gig Ram, 1TB SSD, AvastOmni 20.3.xxx, How to Successfully Install Avast http://goo.gl/VLXdeRepair & Clean Install https://goo.gl/t7aJGq

Offline Lisandro

  • Avast team
  • Certainly Bot
  • *
  • Posts: 67275
Re: The truth about av comparatives? How do they get paid?
« Reply #8 on: November 27, 2011, 08:55:56 PM »
My intention was never to insult any one
I really doubt that he was talking about you, but about them.
The best things in life are free.

Offline Rednose

  • Pirate Party Member
  • Avast Überevangelist
  • Massive Poster
  • *****
  • Posts: 3624
  • Bits of Freedom : https://www.bof.nl
    • Nederlandstalig Avast! forum
Re: The truth about av comparatives? How do they get paid?
« Reply #9 on: November 27, 2011, 09:07:21 PM »
Interesting that Comodo don't want to have the results be published. Because I remember that before Melih was banned from Wilders, he made a statement there he had no problem with testing results to be published.

Greetz, Red.
OS: Win 7 x64 SP1 / Ubuntu / Qubes OS / iOS
Real Time: Avast Premier Beta + AMS for iOS Beta WinPatrol Plus Unchecky MCShield  HOSTS File: MVPS + MDL
On Demand: MBAM SUMo
Backup: Win 7 Image
Proxy: ASL VPN's Socks 5 Tor


Offline Gargamel360

  • Avast Evangelist
  • Super Poster
  • ***
  • Posts: 2347
  • Memento Mori
Re: The truth about av comparatives? How do they get paid?
« Reply #11 on: November 27, 2011, 09:51:48 PM »
Mileh ends up with his foot in mouth again, eh?  Not surprised, he is clearly the Rev. Al Sharpton of the computer security field. 

I mean, come on....even if you use and love Comodo products you have to admit, Mileh's mouth has a habit of writing checks his facts can't support.

At the end of the day....you can either trust the documentation and methods the tester/s (in this case av-comparatives) provide you...or not. 

While I still question the use of the results of such tests in such a rapidly changing threat environment, I never really had any questions about "who the tester is working for", especially in this case.  I trust av-comparatives to give me the best possible broad-based product vs. product tests.
Signature?  But I gots no pen....

Offline Chris Thomas

  • Avast Evangelist
  • Super Poster
  • ***
  • Posts: 1936
  • Christian Geek - aka 'born again' Geek
Re: The truth about av comparatives? How do they get paid?
« Reply #12 on: November 27, 2011, 10:34:08 PM »
Always doubted Melih Abdulhayoglue  ::)

Offline Lisandro

  • Avast team
  • Certainly Bot
  • *
  • Posts: 67275
Re: The truth about av comparatives? How do they get paid?
« Reply #13 on: November 28, 2011, 01:09:27 AM »
Seems now it clearer how things work and who pays for what.
At the end of the day I won't be sad to know that vendors pay a fee.
But I still think, if I'm not wrong, that:
1. *Any* result should be published (in an open test), liking the company or not.
   The company shouldn't have the option to not-publish a public test.
2. No problems on asking for a second fee if the results are published for marketing reasons.
3. Confidentiality agreements are to be followed.
4. Payments should never interfere in the results. If the user receive just *part* of the truth, then the trustfulness in the tester and the professionalism and the independence would be zero (at least for me).
5. Testers should public acknowledge their paid for the vendors to test the products (again: payments should never interfere in the results).
The best things in life are free.

Offline ady4um

  • Massive Poster
  • ****
  • Posts: 2667
Re: The truth about av comparatives? How do they get paid?
« Reply #14 on: November 28, 2011, 04:53:54 AM »
Tech, that list is what IBK explained, so there is no problem, no contradiction.

The only thing that was not clear (in this specific topic, but it does not mean it is not explained somewhere else) was which are the criteria for a company to get rejected from the first starting point from participating in the public tests.
ADD/REMOVE PROGS -> avast -> CHANGE/REMOVE -> REPAIR & REBOOT
Avast! 7 FAQ | FAQ & KB | Docs | Removal Utils | Configure Mail Shield | report FP | License Registration | UNSECURED?