You might like to have a look at the MS info
here, Steps to take before installing SP3.
I recently had to format and reinstall Windows. The disk I have has SP2 on it. I had SP3 as a downloaded folder, backed up elsewhere, and applied it as part of the reinstall, prior to going online.
Worked a treat.
In effect, it is almost a new operating system. You won't actually notice any difference to speak of, but there is absolutely no need to get the updates you have missed prior to the release of SP3, but since SP2. The chances are that SP3 might supersede some of those, anyway.
So what I am saying is, to answer your question, no, it would not be unwise. It would be wise. Do it. Windows will catch up with the changes as required.
You will have a lot of updates to install following SP3, regrettably. Dozens of MB.
There's no real way round it if you want to remain patched, unless you can get a friend to load all of those on to a CD or flash drive, also.