What you are seeing isn't fragmented files, otherwise you would see a list of all fragmented files in the log of XP's Defrag.
There is fragmentation and there is also optimisation where you also organise the files after they are defragmented so the files are at the start of the drive, there are different options for optimisation as well. You can have most frequently used files at the start of the drive and several other options to place files at the start of a disk to reduce disk read head travel.
Files that are fragmented are a greater impact in terms of reduced speed as the disk read head has to flit around the disk many times to read the data. Whilst defragmented and optimised files also only require one movement of the disk read head modern drives won't see much difference between retrieving data from a single location.
So personally I don't thing a user would see much difference as we are talking micro seconds difference to read a file at the start of the disk to one at the end of the disk.
Contig.exe is a defragmentor only not a file optimiser, you could use JKDefrag which I also believe is quicker than XPs and if you use the JKDefragGUI it makes these file optimisation options very configurable in a GUI interface.
