I sympathize with this. My fretting hand is in a world of its own when sweeping.
But going back to the subject at hand, as Jak said, patience is a must. Also, do not neglect what Ids said. Knowing how to mute those bloody strings is quite important. Sure, when recording you can get away with it...