I only have one add-on on the list. Download Helper, however It doesn't seem to impede my internet hopping all that much while it is in progress. The first thing I do before I use it is clear my cache. That helps.
I think that most of the slowing is about the initial firefox startup as it loads these add-ons. Personally I don't see any issue on browsing with those that I have installed and if anything the security and blocking add-ons I have, NoScript, AdBlock Plus and RequestPolicy could help speed browsing due the the cr*p they stop loading or scripts being run, etc.
It will always be a compromise on what you load, the whole point of the add-ons is the way it should help the way that you browse and the additional security that you want. The same as running an AV to protect your system there will always be a performance hit, choosing the right AV (avast of course) limits that hit.