Avast WEBforum
Consumer Products => Avast Free Antivirus / Premium Security (legacy Pro Antivirus, Internet Security, Premier) => Topic started by: hgratt on May 22, 2005, 05:09:33 PM
-
Just a friendly reminder for the Avast team to not forget the problem with severe transmit speed degradation when outbound email scanning is enabled. See the following link for details:
http://forum.avast.com/index.php?topic=13520.0
Thanks,
Harvey
-
Harvey,
I have prepared a special optimized build for you to test it. As we don't see the same network speed slowdown it's a little bit hard to say if this modification would make any difference.
Please download this: http://www2.asw.cz/~rypacek/ashmaisv.zip
Stop Internet Mail Provider and unzip the file into the avast4 folder. Zipped are two files (ashMaiSv.exe and ashmaiSv.exe.sum). The .sum file is there to prevent file replacement during avast! integrity checks.
Then please edit the avast4.ini and add this line to [MailScanner] section:
[MailScanner]
BurstTransfer=1
and start Internet Mail again.
I have made some testings just now at my home DSL connection. Some were pretty nice but sending the mail again and again I got the high speed results for both the old and this modified transfer modes. So, maybe it will make any difference at your setup and maybe not.
Thanks for helping.
Lukas.
-
O.K., did the patch you gave. No difference. With outbound email scanning enabled, I only get around 35-37KBS transfer rate. With outbound email scanning disabled, I get around 75KBS. I should be getting around 92KBS, apparently just having Avast installed is causing a degradation.
What upload speed does your DSL connection have? The 35-37KBS is pretty much near the maximum obtainable if your upload speed is 384kbs. I think you need a much higher upload speed to really see the degradation.
Thanks,
Harvey
-
Harvey, the transfer is done via one system call, TransmitFile(). According to the documentation it is the best possible method of transferring data when they reside in a file (a tempfile after being scanned in this situation). It involves just one transition to the kernel mode and uses file caching to the maximum possibly extend. Of course the network speed is rather slow when compared to the disk IO speed possibilities but with the TransmitFile() in this latest test-build there is nothing left in this area to optimize.
It must be something else!
To complete the testing I suggest you try to uninstall the Network Shield from avast! It is not monitoring outgoing connections (and their content) but no other avast! component is being involved during the SMTP send.
-
Lukas,
in Avast mail scanning are there any "optimised interfaces" for differing mail clients? I have the Microsoft clients in mind as I ask the question. I assume they must feature pretty strongly in your development and testing.
If not, would you expect the scanning and transmission of the same message sent by different clients to be just about the same?
-
Alan, I don't know what exacly do you have in mind. Perhaps Vojtech might give us some in-depth info. But as far as I know there are no optimization for specific mail clients. The POP3 and SMTP interfaces (protocols) are described by their respective RFCs and avast! follows this standard.
Concerning the outgoing scan - it works like this:
1) client connects to the Internet Mail provider (either manualy configured or via redirect)
2) sends the mail via SMTP comamnds
3) Internet Mail stores it into temporary file and scans it
----------< up to here this should be very quick, if I neglect the time for scanning it is all on localhost
4) the Internet Mail provider connects to the SMTP server and starts sending mail
----------< here is the step when your's and Harvey's slowdown is observed
In this latest test-build (as I have written just in my previous message) the step 4 is replaced by single API call - TransmitFile(). This is the function used by IIS to transmit files (on Web server) - a highly optimized method of network transfers. And it does not involve any SMTP commands at all, it is just a plain byte burst on the network interface on a single line of source code. If this is not fast enough someone else is getting into the way...
-
Harvey and I were both conducting our tests using Thunderbird.
I assumed that there would be no difference if I used another mail client to send the same message, but I thought I should test that assumption.
So I repeated the tests using Outlook Express. The results surprised me.
I conducted the same tests multiple times to make sure they were consistent.
Without Avast mail scan OE is about 40% slower in transmitting the same file attachment (pdf) than Thunderbird.
With Avast mail scan the slowdown for OE is much less than the slowdown for Thunderbird and OE becomes the clear winner.
One interesting note. The transfer of the SMTP stream from OE to completion of caching by Avast took less than 2 seconds, the transfer of the SMTP stream from Thunderbird to completion of caching by Avast took 8 -9 seconds.
I will repeat my tests today followng the instructions you gave Harvey to see if I can see any difference.
-
Harvey, the transfer is done via one system call, TransmitFile(). According to the documentation it is the best possible method of transferring data when they reside in a file (a tempfile after being scanned in this situation). It involves just one transition to the kernel mode and uses file caching to the maximum possibly extend. Of course the network speed is rather slow when compared to the disk IO speed possibilities but with the TransmitFile() in this latest test-build there is nothing left in this area to optimize.
It must be something else!
To complete the testing I suggest you try to uninstall the Network Shield from avast! It is not monitoring outgoing connections (and their content) but no other avast! component is being involved during the SMTP send.
I uninstalled Network Shield - no difference. I then uninstalled Avast - still only averaged around 83KBS.
I then re-installed NAV 2004 to test it out again. Amazingly, I was now back to getting around 92KBS!
All I can think is that NAV also fixed some bad registry entries since I'm getting better speeds with it installed than with no AV installed at all!
Sure would like to know what is going on.
Thanks,
Harvey
-
Here are my test results for today.
Same message sent in each test (time in seconds).
Mail Scanner Mail Scanner Mail Scanner
off 4.6.665 Optimized,BurstTransfer=1
ThunderBird 88 134 131
Outlook Express 136 130 128
The numbers for OE are in the correct columns.
-
Lukas and/or Vojtech,
The Avast Internet Mail provider is in the unenviable position of having to emulate both POP3 server and client as well as SMTP server.
I say unenviable because the number of real products you are standing in for on both sides must be considerable.
I understand the comment about compliance with RFC standards made by Lukas, but are you settling for "as good as Microsoft" or are you looking at "best of breed" candidates if another product might have found a more efficient way of getting the job done?
I will certainly understand if you come back and tell us that "it needs to be this way to meet the needs of most clients and most servers" - a kind of lowest common denominator.
Frankly, in the cold light of day, while this is an interesting technical issue for me it is not a burning day to day performance issue for me or for those I support. Most of us do not send enough large messages that even a 100% slowdown in transmission of the messages would warrant any real concern and - to return to an old refrain - especially at the price point many of us pay for Avast.
I do not seek to downplay the real issue this may represent to other Avast users who may be running listservers etc. and who need to optimize mail transmission with the assurance of virus prevention. I will leave this issue to them and the smart minds of the Avast team.
I think I have another issue with the Internet Mail provider I need to devote some research time to.
-
Lukas, it would still be interesting to know what your DSL upload speed is. As I indicated before, I think that anything at 384kbs or lower will mask the problem.
Thanks,
Harvey
-
Unfortunately Lukor is currently on a vacation so I guess I'll follow up.
I think he tested it here with 4mbit uplink.
alanrf is suggesting that the problem does not appear when using OE as email client (that is, with OE, it is always slow - no matter if avast is installed or not). Is that statement valid even on your machine?
Also, do I understand it correctly that you're getting these results even with outbound scanning disabled. What if you uninstall the Internet Mail provider completely - does that make any difference? And other providers...?
Somehow, we still don't know what component is causing the problem, exactly... :-\
Cheers
Vlk
-
Unfortunately Lukor is currently on a vacation so I guess I'll follow up.
I think he tested it here with 4mbit uplink.
alanrf is suggesting that the problem does not appear when using OE as email client (that is, with OE, it is always slow - no matter if avast is installed or not). Is that statement valid even on your machine?
Also, do I understand it correctly that you're getting these results even with outbound scanning disabled. What if you uninstall the Internet Mail provider completely - does that make any difference? And other providers...?
Somehow, we still don't know what component is causing the problem, exactly... :-\
Cheers
Vlk
I have tried it uninstalling the components with the same results. If outbound scanning is disabled and/or pertinent components uninstalled, I still only get about 83KBS (kilobytes/sec) transfer rate (35KBS with Avast outbound email scanning enabled).
Even if I uninstall Avast completely, I still only get about 83 KBS. As I indicated in a previous post, I now have re-installed NAV 2004 and, amazingly, my outbound email speeds (with NAV outbound scanning enabled) is back up to 92KBS.
This would seem to imply that the NAV installation either fixed some bad registry entries created by Avast and/or NAV somehow handles transmissions more effectively. My guess is some registry settings (or equivalent) were "fixed" since I always get around 92KBS upload speeds when I test with various speed sites regardless of what AV is installed.
This is a very strange situation.
Also, FWIW:
1. I am using the Mozilla 1.7.8 email client
2. Several other individuals on a Comcast site also see the same problem.
I appreciate your looking into this.
Thanks,
Harvey
-
I tried to simulate it here (with ThunderBird) but don't see any considerable slowdowns.
I'm measuring only the time from the Thunderbird progress bar reaching 100% to its closing - it's practically the same as with outbound scanning disabled (on this test machine).
How fast does the progress bar go on your machine? It should go really fast because it is transferring data only to the localhost avast proxy.
BTW
seem to imply that the NAV installation either fixed some bad registry entries created by Avast
I doubt that because that would mean that after you actually uninstall NAV, it would work just as fast.
Anyway, IMHO differences < 10% can be caused by just about anything...
-
I tried to simulate it here (with ThunderBird) but don't see any considerable slowdowns.
I'm measuring only the time from the Thunderbird progress bar reaching 100% to its closing - it's practically the same as with outbound scanning disabled (on this test machine).
How fast does the progress bar go on your machine? It should go really fast because it is transferring data only to the localhost avast proxy.
BTW
seem to imply that the NAV installation either fixed some bad registry entries created by Avast
I doubt that because that would mean that after you actually uninstall NAV, it would work just as fast.
Anyway, IMHO differences < 10% can be caused by just about anything...
Need to make sure we're talking about the same thing.
When scanning is enabled, I first see a progress bar which, I believe, indicates the time being taken for the scanning to be performed (this process is relatively quick).
After that bar reaches 100% (I assume that the scanning is now completed), the actual outbound transmission starts up. It is then that I monitor my speeds via NetPerSec (and a stopwatch) - these are the speeds that I am reporting.
Thanks,
Harvey
-
Allrighty, here's another version for you to test - I made some more changes to the code.
http://www2.asw.cz/misc/ashmaisv.zip
The installation procedure of the patch is the same as described above (you don't need to set anything in the INI file this time).
Please let me know how it goes. ;)
Thanks Vlk
-
My xp setup now uses NAV. I am reluctant to uninstall it at this time. However, I have a
win98se setup which I could try. I have noticed though, that the redirect tab for the Mail Service is greyed out. Is Avast outbound scanning active in win98se?
Let me know if I should try your patch on the win98se setup. Also, since I will have to reinstall Avast, and then the patch, should I put the lines back in the .ini file?
Thanks,
Harvey
-
I'd say it's still worth trying on the Win98 machine, although for the sake of candour it would be best to do the test on the XP box (please please, it's just like ten minutes isn't it? ;D).
No INI changes are necessary.
-
O.K., I tried it. Here are the results.
1. After uninstalling NAV 2004 and before installing Avast, I tried another test. My outbound speeds dropped from 92KBS to around 80-83KBS! Consistent with what I've seen before, but still a mystery.
2. Your patch did not fix the problem.
3. Your patch made my system unstable.
If you don't have any other suggestions, I will reinstall NAV. Please let me know as soon as you can if there's something else to try.
Thanks,
Harvey
-
2. so it's still at 3x KB/s? Quite strange. On my test machine, the patch changed the speed from 220 KB/s (speed with Internet Mail provider disabled) to around 550 KB/s -- it actually more than DOUBLED the upload speed...
3. What exactly does that mean?
Thanks
Vlk
-
With scanning enabled, the mail client seemed to stop responding (it still sent mail). The first time I tried it, it froze NetPerSec. I then rebooted and tried again - only got about 35KBS. Then when I tried to save the test mail, mozilla hung.
Thanks,
Harvey
-
...must've been because of the 100% CPU usage the new ashMaiSv.exe was using during data transfer (because of its attempt to maximize the throughput).
Here's the last version: http://www2.asw.cz/misc/ashmaisv2.zip -- if that doesn't help, I must be dreaming (because it IS changing the speeds here - and very noticably).
Thanks for your help (and I'm really sorry for the hassle - I'm getting the impression that you're now incllining to NAV anyway... :-\).
Cheers
Vlk
-
...must've been because of the 100% CPU usage the new ashMaiSv.exe was using during data transfer (because of its attempt to maximize the throughput).
Here's the last version: http://www2.asw.cz/misc/ashmaisv2.zip -- if that doesn't help, I must be dreaming (because it IS changing the speeds here - and very noticably).
Thanks for your help (and I'm really sorry for the hassle - I'm getting the impression that you're now incllining to NAV anyway... :-\).
Cheers
Vlk
O'K., this IS different!
1. With scanning disabled, I still only get around 80KBS.
2. However, now, with scanning enabled I am averaging close to the 92KBS that I see with NAV 2004. However, the speed as a function of time is very "spikey" (using NetPerSec) - that is, I will see lows of around 30KBS followed by highs of around 134KBS. The average appears to be around 92KBS.
This is worth further experimenting with.
Can you elaborate on what you did and why it is working as it is?
Thanks,
Harvey
-
...must've been because of the 100% CPU usage the new ashMaiSv.exe was using during data transfer (because of its attempt to maximize the throughput).
Here's the last version: http://www2.asw.cz/misc/ashmaisv2.zip -- if that doesn't help, I must be dreaming (because it IS changing the speeds here - and very noticably).
Thanks for your help (and I'm really sorry for the hassle - I'm getting the impression that you're now incllining to NAV anyway... :-\).
Cheers
Vlk
O'K., this IS different!
1. With scanning disabled, I still only get around 80KBS.
2. However, now, with scanning enabled I am averaging close to the 92KBS that I see with NAV 2004. However, the speed as a function of time is very "spikey" (using NetPerSec) - that is, I will see lows of around 30KBS followed by highs of around 134KBS. The average appears to be around 92KBS.
This is worth further experimenting with.
Can you elaborate on what you did and why it is working as it is?
Thanks,
Harvey
Well, I spoke too soon. After rebooting back into win xp, I recieved a message indicating Avast had to reload my OS. I rebooted and now with scanning enabled, I'm back to the same 35KBS speeds.
Harvey
-
Oh yeah, the auto-updater replaced the patched file with its original version (as a security measure).
What version of avast do you have installed, exactly? Do you have the latest one - 4.6.665?
Thanks
Vlk
-
Oh yeah, the auto-updater replaced the patched file with its original version (as a security measure).
What version of avast do you have installed, exactly? Do you have the latest one - 4.6.665?
Thanks
Vlk
Wait, wait, wait. I thought I had the latest version - but I have 4.6.652. I redid the patch and I was back to the 92KBS (erratic) average, so the reboots must have clobbered the changes I will update to the latest version and redo the patch and get back to you.
Thanks,
Harvey
-
Vlk,
could not resist trying the lastest patch you offered to hgratt.
Interesting -
My test message has varied in delivery time only between 87-90 seconds with Avast turned off, well within the variations of Comcast - my (and hgratt's) cable provider.
With the lastest patch, from hitting send to completion the result was 100 seconds.
Allowing for the fact that I have reported before that it is taking 8-9 seconds to create the temporary cache (in the case of Thunderbird) and less than two seconds in the case of OE, it seems to me that you have just about matched the transmisson speed of Thunderbird.
-
hgratt: OK, let me explain what's up.
The ZIP file (the patch) contains two files - a .DLL and a .DLL.SUM.
The .DLL is the patch itself (the binary). The .DLL.SUM is a special file that is telling the avast updater that this is an authorized patch (otherwise, it would get replaced by the original version - this is part of the defensive strategy built-into avast to fight with unauthorized changes caused by malware).
However, the authorization is bound to a specific version number. This is because when a new version is installed, we want the patch to be updated by a new version.
And, the .DLL.SUM file in this particular ZIP file is for version 4.6.665 (so you need to have this version installed for the DLL.SUM file to be effective).
Hope this helps,
Vlk
-
Vlk,
the result for the same message when using OE was 92 seconds compared with 130 seconds yesterday.
So, you have made sending mail by OE scanned with Avast much faster than sending it without scanning ... you should sell it to the folks in Redmond!
-
;D
There's really no rocket science here. I changed only two things:
1. increased the chunk size (the size of data sent at once) from 8KB to 128KB
2. removed the sleep command after sending each chunk.
That is, there were artificial sleeps (delays) implemented in the code - I don't know why yet (I'll have to ask the developer responsible for this code tomorrow).
It's possible they were there for a reason - but I'd say the main reason was to prevent excessive CPU loads for the ashMaiSv.exe process...
Thanks
Vlk
-
Hmmm ... just in line with what I suggested was happening in the original thread Harvey created on this issue.
Thanks for being upfront about the issue - it is appreciated.
-
Just one more thing, can someone check the difference in the creation of the temporary cache file between Mozilla code and OE?
Unless there is some oddity in the Mozilla code, it should not take 8 seconds to cache the same amount of data from Mozilla as it takes less than 2 seconds from OE.
-
;D
There's really no rocket science here. I changed only two things:
1. increased the chunk size (the size of data sent at once) from 8KB to 128KB
2. removed the sleep command after sending each chunk.
That is, there were artificial sleeps (delays) implemented in the code - I don't know why yet (I'll have to ask the developer responsible for this code tomorrow).
It's possible they were there for a reason - but I'd say the main reason was to prevent excessive CPU loads for the ashMaiSv.exe process...
Thanks
Vlk
I updated the program and it now works on subsequent reboots. Looks like we're making progress. Again, the speed averages around 90-92KBS, but is quite erratic.
Based on your explanation as to what you did, it sure seems that many people should see this issue - not just alanrf and myself.
Anyway, at least the area of concern seems to be identified. Maybe some further optimization will "smooth" out the upload speed.
Thanks,
Harvey
-
Erratic? Or just uneven?
-
Erratic? Or just uneven?
Uneven. Speed bounces back and forth from lows of 30KBS to highs of 135KBS.
Harvey
-
Isn't it a bit strange? I mean, considering that the maximum speed of your line is < 100KB/s (physically), isn't it a problem in the way NetPerSec does the measurements?
I was using LanSpeed2 to check the speed and I didn't see any anomalies...
Thanks
Vlk
-
Isn't it a bit strange? I mean, considering that the maximum speed of your line is < 100KB/s (physically), isn't it a problem in the way NetPerSec does the measurements?
I was using LanSpeed2 to check the speed and I didn't see any anomalies...
Thanks
Vlk
I think only the average speed is constrained.
Do you have a copy of LanSpeed2 that I could try?
Harvey
-
http://www.tucows.com/get/259303_100243
-
http://www.tucows.com/get/259303_100243
O.K., I set the interval for 1 sec (same as NetPerSec) and the line graphs show the same uneven behavior and same average speed.
BTW, I really appreciate all the time and effort your putting into this. Hopefully, this will benefit everyone.
Harvey
-
I'd say it's because ashMaiSv.exe is yielding to other processes for a short while after sending each chunk of data (128KB)...
I don't think there's a problem.
-
Based on your explanation as to what you did, it sure seems that many people should see this issue - not just alanrf and myself.
Harvey,
I think the important word is see it.
It looks to me as though the base Mozilla code has adopted a
transmission method that delivers mail to the network faster than some
other products, most notably those of Microsoft.
I suspect that everyone using Mozilla based mailers is being affected by
this issue. However, the Microsoft products are much more widely used than the Mozilla based ones.
Another part of "seeing" it is the interface.
In Outlook Express as soon as you hit send the message window disappears (therefore it must have been sent). There is a little icon on the main OE window showing mail is being transmitted but I doubt that many people are going to time it.
I don't know about all Mozilla mailers but Thunderbird has a progress
bar showing the progress of the message delivery. I do know a number of
Thunderbird users who have turned off the outbound scanning of e-mail
because the change in behaviour of the progress bar caused by Avast
disturbed them. People expect progress bars to move fairly consistently
to 100% and then say "done". When the bar moves rapidly to 100% and
then sits there it generally causes disquiet. (This has disturbed users of Avast on the receiving side of other applications too - notably with NNTP).
Finally I suspect most concern comes when folks are waiting for something
to arrive to them rather than when they are sending and do not expect to
see the final event (the delivery of the message to the person(s)
addressed).
Just my thoughts.
Anyway, you were the one who noticed this issue and raised it and I think your efforts will benefit all of us. Well spotted!
Oh, and didn't I tell you the Avast team would make sure you didn't need
to go back to NAV?
Alan
-
I'd say it's because ashMaiSv.exe is yielding to other processes for a short while after sending each chunk of data (128KB)...
I don't think there's a problem.
I just tried task manager in order to give ashMaiSv.exe high priority to test out what you said, but I was denied access (I can set priorities on other processes).
Anyway to get access so I can fool around with the priority?
Harvey
-
Increasing the process priority wouldn't help anyway (it is explicitly yielding).
I'll do some more tweaking tomorrow and will let you know (tomorrow)...
Thanks
-
Increasing the process priority wouldn't help anyway (it is explicitly yielding).
I'll do some more tweaking tomorrow and will let you know (tomorrow)...
Thanks
Thanks, Vlk.
Again, I appreciate all you are doing.
Harvey
-
Harvey,
just for future reference to get around the "access denied" problem should you need it.
Choose a time (hh:mm) a couple of minutes in the future.
Open a command prompt window, type and enter:
at hh:mm /interactive taskmgr
At the time you chose hh:mm a taskmgr window will open and should allow you to change the priority if you wish.
-
Or download http://www2.asw.cz/~vlk/er.exe (my own tool :)) and (with Task Manager NOT started) run
er taskmgr.exe
This will also start the Task Manager with elevated rights (if you're logged on as an admin)
Cheers
Vlk
-
I was able to change the priority to High, but it was still uneven.
Worth a try.
Thanks,
Harvey
-
OK folks, here's another version: http://www2.asw.cz/misc/ashmaisv3.zip .
This time, the behavior should change if you add the following line to the [MailScanner] section of the avast4.ini file:
SendInBlockingMode=1
This should result in maximum throughput with no spikes... (hopefully)
Thanks :)
Vlk
-
OK folks, here's another version: http://www2.asw.cz/misc/ashmaisv3.zip .
This time, the behavior should change if you add the following line to the [MailScanner] section of the avast4.ini file:
SendInBlockingMode=1
This should result in maximum throughput with no spikes... (hopefully)
Thanks :)
Vlk
Vlk,
Much better! This is comparable to what I was seeing in NAV 2004. In fact, the average speed may even be slightly higher.
Could you explain what this latest fix does? I hope they give you stock options!!!
Thanks,
Harvey
-
Good! :)
Could you explain what this latest fix does?
Well, what this patch does is replace a piece of lousy code with a good one ;D. No, really, I have to admit that the actual sending code was a bit strange, to put it diplomatically. It was written in +-1998 and it's likely that our coding standard was a bit lower at these days ;). Oddly enough, the code WAS working after all, but sort of suboptimally.
I hope they give you stock options!!!
You bet! ;)
Cheers
Vlk
-
Probably explains why those of us using Stunnel for the actual transmission didn't have a problem.