Author Topic: RAM footprint question  (Read 16276 times)

0 Members and 1 Guest are viewing this topic.

denro

  • Guest
RAM footprint question
« on: November 24, 2003, 12:38:36 AM »
I have avast running with resident file scanning and email scanning.  The p2p was not installed.  I had heard that the memory use of avast was modest compared to Norton (2002) anti-virus.  But in opening up the Task Mgr in Win XP I noticed that avast has 4 programs running resident taking up total memory of about 38 Mb.  When I was using Norton AV, background scan for files and mail, there were only two programs resident for a total of about 6 Mb.  What am I missing?  Is it true then that avast takes about 6 times the memory as does Norton AV?  That seems fairly heavy useage.  Any comments?  Otherwise, avast appears to be an awsome program.

stevejrc

  • Guest
Re:RAM footprint question
« Reply #1 on: November 24, 2003, 01:21:14 AM »
Maybe Norton was using more virtual memory or something, or its not as comprehensive as Avast. Might depend on how high you set the detection rate. Avast picks up viruses better than Norton in tests so maybe the memory usage reflects its superior detection rate.

Do you use home edition, how did you not install p2p, I didnt think it was optional? I have all 5 components installed.

denro

  • Guest
Re:RAM footprint question
« Reply #2 on: November 24, 2003, 01:24:49 AM »
stevejrc, p2p can be deselected during install.  I am still surprised to see avast taking up so much more memory than avast.  I am running XP Home.

kmolazz

  • Guest
Re:RAM footprint question
« Reply #3 on: November 24, 2003, 01:43:56 AM »
I've just started using avast 4 home, and also noticed that it uses a lot of ram.  Right now i only have the standard shield and the mail filter, and together with "ashdisp.exe" service, they sum up to 33MB. Is this normal? I come from using AVG which i now uses very little resources... but i though avast didn't use that much ram...

Offline Lisandro

  • Avast team
  • Certainly Bot
  • *
  • Posts: 67194
Re:RAM footprint question
« Reply #4 on: November 24, 2003, 03:04:45 AM »
Hey brothers... If an application does not use RAM what will it do? RAM consumption were discussed in so many past foruns...

You can configure avast! at a low level (disabling the residents you do not use, disabling 'throghout' options of scanning like 'extensions', and so on...). avast! runs smootly in a XP system with 256 Mb RAM for me  ;)
The best things in life are free.

Offline igor

  • Avast team
  • Serious Graphoman
  • *
  • Posts: 11849
    • AVAST Software
Re:RAM footprint question
« Reply #5 on: November 24, 2003, 09:42:12 AM »
As mentioned many times on this board - you should check the "Virtual Memory Size" column in the Task manager - not "Memory Usage" (which means something else than most of the people expect).

Hornus Continuum

  • Guest
Re:RAM footprint question
« Reply #6 on: November 24, 2003, 08:52:51 PM »
Trying to determine exactly how much memory an application uses is extremely diificult, requiring voodoo science some would say.  Part of the problem is ambiguity in the terminology, as illustrated below.

The Task Manager help file defines Memory Usage as the Working Set for a process, the number of pages currently present in memory, in KB.  It defines Virtual Memory Size as the amount of virtual memory or address space committed to a process.  Paged Pool is defined as the amount of memory charged to a process that can be moved to the pagefile; Nonpaged Pool is operating system memory that is never moved to the pagefile.  Both are measured in KB.

The Microsoft Management Console help file defines Working Set as the current number of physical memory bytes used by or allocated to a process.  The help file goes on to explain that part of that allocation may be located in the pagefile.

However, something is wrong with these definitions.  First, notice the contradiction in the definition of Working Set.  Start Task Manager and look at the Processes tab; you'll notice some glaring inconsistencies with these definitions.  First, if they are correct, then the amount of memory listed for a process'es Virtual Memory Size would always be greater than or equal to the amount listed for its Memory Usage, but you'll see many cases where MU exceeds VMS.  Second, look at the numbers for Paged Pool and Nonpaged Pool.  They are all listed in tens or hundreds of KB's, leading me to believe that the figures listed are the amount of memory used by the processes' table entries in the pools, not the actual amount of memory.  Notice also that some processes are listed as having 0 KB in both the Paged and NonPaged Pools, yet thir Memory Usage and Virtual Memory Size are non-zero.

Another big part this problem are the definitions of 'virtual memory' and 'allocated memory.'  Seemingly everybody has a different concept of their meanings.  First, some say virtual memory is that part of a program's memory allocation that resides in the pagefile, others that it's the sum of the amounts of its physical memory and that saved to disk.  The former is conceptually close enough for most discussions, but the latter is the actual truth.  Second, most people think allocated memory is the memory that a program is actually using; however, it is the memory that has been made available for it to use, but some of that memory may actually not be in use at a given time, as I'll explain later.

Programs do not work with physical memory; they work with virtual memory.  When an application starts, the OS assigns it a virtual address space; the address range is irrelevant, but it's is much larger than the amount of memory the program will ever use.  The size of the application's virtual address space is the amount of memory currently allocated by the OS for the program to use.  For the remainder of the discussion, I'll use the term 'address space,' but realize that this refers to virtual address space.  Also, be aware that the terms 'allocated memory' and 'memory allocation' also refer to memory in the virtual address space.

Based on the requirements specified in the executable, some of that address space is put to use immediately, allocated by the OS from physical memory to load the program executable, core DLL's, and initial program stack and mapped into the address space.  It also allocates a memory segment for use as a memory heap.  During it's initialization, the program may request more memory, but depending upon the nature of the request(s), some of that memory may be allocated from physical memory and some from the pagefile.  (Some programs request more memory than they need, in anticipation of that need -- a performance improving practice carried over from the DOS and Windows 3.1 days.  There's little to nothing to be gained from this with modern Windows OS's.)

When the program first accesses a portion of its allocated memory that exists only in the pagefile, e.g., to load a DLL for a spell checker or for use as a file I/O buffer, one or more page faults are generated.  In response, the OS with the aid of the CPU's memory manager maps the required number of page(s) of physical memory to a portion of the program's address space, making them available to the program and increasing the size of its allocated memory.

An example of a memory allocation that exists only in the pagefile that avast! users can identify with occurs when an application opens a file for writing, but the Behavior Blocker intercepts the attempt, and the user denies permission.  Memory for an I/O buffer is allocated for the program, but until the I/O operation occurs, it is never used and resides on the hard drive.

Now, consider a user who is surfing the Web and has a dozen windows open, each displaying pages from different web sites.  The browser has requested memory to display those pages.  Assume for the moment, the user is actively moving between 2 of them, and all the memory used for these windows is located in physical memory.  Then, the user responds to a virus database update and starts a thorough scan of the hard drive including archives, and the drive contains a lot of large Zip files.  Let's also assume this user has a pitiful machine with only 64 MB of RAM.  As avast! requests and receives more and more memory, increasing the size of its address space, and the amount of free physical memory drops lower and lower, the OS "takes away" some memory from the browser in order to meet the scanner's requirements.  Because the OS tracks physical memory usage and is aware that the memory holding 10 of the dozen web pages hasn't been accessed for awhile, it swaps the contents of some of it to the hard drive and maps the physical memory into avast!'s address space.  While the size of avast!'s memory allocation increases, the size of the browser's doesn't decrease; it gets no smaller, but some of it's contents are now located in the pagefile.

At this point, the user continues surfing the web and clicks a link in a third browser window.  Since the contents for this memory are on the hard drive, one or more page faults are generated.  At this point, the OS swaps the contents of some of the physical memory used by avast! to the pagefile replacing it with the contents needed by the browser and maps it into the browser's address space.  The memory allocation for both programs remains unchanged, only the location of their contents changes.  Meanwhile the on-demand scanner, which is running in the background, tries to access the contents of its memory now held on the hard drive.  The OS swaps out the contents of the physical memory that the scanner needs the least and replaces it with the data that it needs now.  Again, the size of the address space for both programs remains unchanged.

At this time, the user notices that the virus scan is taking a real long time and closes the windows for the 9 unused web sites, and the browser releases the memory back to the operating system.  The size of the browser's address space decreases.  The next time the scanner generates a page fault, the OS replaces the contents of that page with the correct page from the hard drive.  The browser's memory allocation drops, but the scanner's remains unchanged.  But because the browser has released some memory, if avast! requests more memory to process the biggest Zip file seen so far, the OS allocates it from physical memory -- no swapping required.

So what's the upshot of all this?  I can't remember at this point.   ;D  Oh, yeah, simply looking at the numbers spit out by Task Manager is not enough.  Microsoft's definitions conflict with each other, and the information displayed by Task Manager apparently doesn't always match the definitions, nor does it always make sense.  The impact of avast!'s memory usage is not only determined by the size of its memory allocaton, but how it uses it in conjuction with everything else running on the system.  The bottom line question isn't how much memory does avast! use, but how much does this usage negatively impact your system, and can that impact be reduced?  Getting the answers requires a detailed analysis.  The Performance Monitor can aid in this quest, and its help file has quite a but of information and advice for going about it.

Regards,
Hornus
« Last Edit: November 24, 2003, 09:18:42 PM by Hornus Continuum »

denro

  • Guest
Re:RAM footprint question
« Reply #7 on: November 25, 2003, 12:00:42 AM »
Thank you Hornus for the excellent and detailed reply.  I am keeping a permanent copy of your response.  It is worth some study while watching the various available columns of the Task Manager.  Thanks for your hard work putting this together.

-- Denro

Culpeper

  • Guest
Re:RAM footprint question
« Reply #8 on: November 25, 2003, 12:31:41 AM »
Yes, thanks for the input!  We should use this as a standard response (with our source cited of course :D) every time this issue comes up.  And it comes up often enough.

Hornus Continuum

  • Guest
Re:RAM footprint question
« Reply #9 on: November 25, 2003, 12:41:13 AM »
Thanx for the kudos guys.  But think about all the bandwidth that would be consumed.  Do you think the Internet could handle it?  :D

Regards,
Hornus
« Last Edit: November 25, 2003, 12:41:47 AM by Hornus Continuum »

Offline Vlk

  • Avast CEO
  • Serious Graphoman
  • *
  • Posts: 11658
  • Please don't send me IM's. Email only. Thx.
    • ALWIL Software
Re:RAM footprint question
« Reply #10 on: November 25, 2003, 12:45:15 AM »
Hornus, thanks for the nice reading... again.
I have a couple of comments

Quote
(Some programs request more memory than they need, in anticipation of that need -- a performance improving practice carried over from the DOS and Windows 3.1 days.  There's little to nothing to be gained from this with modern Windows OS's.)

I don't quite agree - in many cases it is really advantegous to grab a bigger piece of memory at once, and then use a custom sub-allocator to partition it into actual memory blocks. That's because the system allocator is of course as generic as possible, providing poor performance in certain situations where a simple custom allocator would in fact provide a big performance boost. Look at the behavior of many server-based apps - they allocate huge memory blocks at startup, and it's (usually) really for a good reason. The extreme example of this is the MS SQL Server which (by default) almost always takes about 85% of free RAM at once on startup ( even if your machine had 4G of RAM, SQL would grab about 3.5G at startup).

Quote
An example of a memory allocation that exists only in the pagefile that avast! users can identify with occurs when an application opens a file for writing, but the Behavior Blocker intercepts the attempt, and the user denies permission.  Memory for an I/O buffer is allocated for the program, but until the I/O operation occurs, it is never used and resides on the hard drive.

Well in fact the EXE images are implemented via memory-mapped files, not ordinary buffer reads. Thus no buffers need to be preallocated (but virual address space need to be reserved [not commited] of course).

Quote
So what's the upshot of all this?  I can't remember at this point.    Oh, yeah, simply looking at the numbers spit out by Task Manager is not enough.  Microsoft's definitions conflict with each other, and the information displayed by Task Manager apparently doesn't always match the definitions, nor does it always make sense.  The impact of avast!'s memory usage is not only determined by the size of its memory allocaton, but how it uses it in conjuction with everything else running on the system.  The bottom line question isn't how much memory does avast! use, but how much does this usage negatively impact your system, and can that impact be reduced?  Getting the answers requires a detailed analysis.  The Performance Monitor can aid in this quest, and its help file has quite a but of information and advice for going about it.

I absolutely agree with this. Memory usage is just one (and probably not the most important) parameter that needs to be considering when rating the performance of a program. But since excessive paging can really have catastrophic consequences in terms of overall system responsiveness (as MS said, 'A page fault can ruin your day' :)), the system, in conjunction with reasonably designed applications, is trying to prevent it as much as possible in most of the cases.


---
Anyway, I wanted to say that we've changed some allocation patterns in the upcoming avast update - check out the Mem Usage column as soon as you get the update... (can't tell exactly when it is gonna be released though) :)

Vlk
If at first you don't succeed, then skydiving's not for you.

Hornus Continuum

  • Guest
Re:RAM footprint question
« Reply #11 on: November 25, 2003, 03:11:34 AM »
Vlk,

Quote
Hornus, thanks for the nice reading... again.

You're quite welcome. I hope you enjoyed reading it as much as I did writing it.  Software engineering is my passion, but unlike most SWE's, I enjoy the technical writing aspects as much as the development aspects.   Most that I know, hate it with such a fervor that they would gladly sacrifice a testicle if it would get them out of writing documentation.  :D  Having been involved in virtually all aspects of the product life cycle, I have an exteme appreciation for the job you guys do.

Quote
I don't quite agree - in many cases it is really advantegous to grab a bigger piece of memory at once, and then use a custom sub-allocator to partition it into actual memory blocks. That's because the system allocator is of course as generic as possible, providing poor performance in certain situations where a simple custom allocator would in fact provide a big performance boost.

You're absolutely right here.  By custom tuning the memory management algorithm for the application, fragmentation and the number of OS calls can be greatly reduced and provide other efficiencies.

When I referenced the good old DOS days, I was thinking of the situation where basically only one application could run at a time, with the exception of TSR's.  Since the OS, device drivers, the extended and/or expanded memory managers, which were basically OS extensions, and most of the TSR's got what they needed up front during bootup, any of the 640K memory space available after the program loaded could and should be freely used without any other considerations to allocate memory blocks and load and unload overlays.

As for the Windows 3.1 environment (and I should have written 3.x), I was thinking of Real Mode and Standard Mode, which didn't support virtual memory) and the early days of Enhanced Mode, which did.  The average software engineers were either too ingrained with past experiences, or having only a rudimentary unstanding of the concepts, couldn't use it effectively to the benefit of whole computing environment.  The better ones designed applications to grab as much memory as the anticipated needs dictated knowing that other applications couldn't be relied on to get and release memory on an as-needed basis.  (It was better to make the other guy sweat an out of memory condition.  :)  )  And of course, they were trying to boost their own program's efficiency by eliminating the overhead of acquiring and releasing memory, especially the context switching involved everytime an OS call was made that required changing the processor to and from Protected Mode.

Quote
Look at the behavior of many server-based apps - they allocate huge memory blocks at startup, and it's (usually) really for a good reason. The extreme example of this is the MS SQL Server which (by default) almost always takes about 85% of free RAM at once on startup ( even if your machine had 4G of RAM, SQL would grab about 3.5G at startup).

I wasn't taking into account servers and server applications, which have a totally different set of requirements and rules of engagement as it were, and I'm glad you pointed that out.  Your example of a server running MS SQL is an excellent example.  Any server, or cluster, operating as a database server is for all intents and purposes running a dedicated task, file and print servers, web servers, and mail servers not so much.

As such, and seeing first-hand the excellent engineering in the home edition, I know that Alwil considers it necessary to go to the considerable trouble and expense of developing separate products for networks and servers to best meet the unique requirements of it's customer's many different environments.  Kudos for taking that route instead of producing a jack-of-all-trades and master-of-none.

Quote
Well in fact the EXE images are implemented via memory-mapped files, not ordinary buffer reads. Thus no buffers need to be preallocated (but virual address space need to be reserved [not commited] of course).

I didn't know this.  You've given me something tasty to dig into.  I'm always :) when someone can clarify an issue for me or correct a misunderstanding.

Quote
Anyway, I wanted to say that we've changed some allocation patterns in the upcoming avast update - check out the Mem Usage column as soon as you get the update...

That's good news for all of us.  Many thanx to you and the rest of the the A-team for continually striving to improve an already excellent product, especially with much of it based on user feedback.

Regards,
Hornus
« Last Edit: November 25, 2003, 03:33:58 AM by Hornus Continuum »

Culpeper

  • Guest
Re:RAM footprint question
« Reply #12 on: November 25, 2003, 03:44:07 AM »
Just out of curiosity I took a current snapshot of memory use using TaskInfo2003

Avast Antivirus Service:
InMemKB                      6640
PrivateKB                      6412
TotalKB                         17,796

Avast Email Scanner:
InMemKB                       4616
PrivateKB                       3988
TotalKB                          15,068


Avast is set up to run Internet Mail and Standard Shield.

Hornus Continuum

  • Guest
Re:RAM footprint question
« Reply #13 on: November 25, 2003, 03:54:13 AM »
Culpepper,

For curiosity's sake, will you take a snapshot of Task Manager's window showing all the memory-related data and post it for comparison?

Regards,
Hornus

Culpeper

  • Guest
Re:RAM footprint question
« Reply #14 on: November 25, 2003, 05:32:56 AM »
I wish I could but I use Win98.   :-\  I don't believe it has a task manager like Win2000

I really even don't understand TaskInfo's finding above.  I don't know the difference betweein InMemKB, PrivateKB, and TotalKB.

Never mind.  See my next post.

« Last Edit: November 25, 2003, 05:49:05 AM by Culpeper »