NeilT,
on the face of it I would have to agree that this does not sound like it is an issue with the time taken to retrieve a large email attachment. However, we still do not know for sure what is happening.
It might prove useful to create (for a while) a more detailed avast! log of what is happening with your mail connections.
You can get the mailscanner to log your connections by editing the avast4.ini file (in Program Files\Alwil Software\Avast4\DATA folder).
In the section headed:
[MailScanner]
add the line:
Log=20
and save the updated file.
The log will be in Program Files\Alwil Software\Avast4\DATA\log\ashmaisv.log
It may help to share the file with us but you should edit the file to ensure that any personally identifiable information is obscured if you choose to post it.