Dispersion, in direct fire, can be atributed to velocity variation (projectile comes out faster or slower) as well as non-repeatable gun repositioning.
If you were to take a sufficiently rigid barrel, put it in a vise, and ensure that the propellant produces the same reaction (along with things like constant friction within the barrel, etc), then a higher velocity projectile from that setup will be more accurate.
That is because the exsternal ballistics it experiences will not be the same. For a set rifling, the higher velocity projectile spins faster, the amount of time it is subject to atmospheric conditions is less, etc.
By the way, rifling and spin CAN be too much. You want just enough to stabilize the projectile. Too much has an adverse effect just like too little. It will also wear out faster.