My Ko and my Le had this problem, all Ko's had the problem (Cheers Dan Ko ). My Le just had one brake mount to high so it must of been just that frame. Is your wheel all the way back in the drop outs?
If I were you I would replace the bars, stem and forks first. They are really heavy and have stupid geo. That will improve your performance more then wheels will