The task manager can be somewhat inaccurate, as one of the Alwil team mentions the Mem Usage in particular, he believes the VM Size is more accurate. Edit: looks like he has paid us a visit
Personally I don't worry about it too much I have much about avast, there are greater resource/RAM hogs at work, like Firefox 3.0 (which ironically is meant to be better at memory management than 2.0) at 100MB; Explorer, My Firewall, etc.
I have had my system up all day and currently avast isn't using much whilst I'm browsing.
Why aren't you using the Web Shield and Internet Mail providers, you can see from my image they don't use much ?