As far as I'm aware, it is related to the Transient and Persistent caches, used to speed up scans. If you defrag and delete that journal, I believe you fill find the next scan takes longer as you have to build up the persistent cache again, I don't know if that is related/located in the journal.
It is certainly the case after defragging with Puran I find the next on-demand scan is a little slower, so I don't know if this is simply because files may well now be in different locations but the file hash would presumably be the same. Or could it be that it is defragging the journal.
The one thing I do know even if the journal is the reason and avast is creating it own journal entry, its size would be minimal even if fragmented when the rest of your hard disk has been defragged.
So I don't think there would be any system performance impact through it not being defragged, only it would seem avast.