Dude, you have GOT to have a lot more patience with yourself. People spend years working on quality sweep picking.
I second this for sure - it doesn't give it's self up easily. Patience is key
Going purely from what you've said, it sounds as if you are possibly playing with your left hand wrist, slightly 'over bent'. When you talk about not being able to sweep back down, this seems like undue tension in your wrist/hand, along with the over bent wrist angles, causing things to lock up.
Here are a few things to try/consider:
1. Sit with your Guitar and have your left hand on the neck, as relaxed as it would be playing a G chord - this is what you want for Sweeps (at least the non stretchy ones) .Because sweeping (at any tempo) relies on your fingers being able to move large distances freely, you want to be relaxed & controlled.
Any undue tension in your playing will be a spanner in the works of the fluidity you need. If you notice yourself tensing as you want to sweep faster, it's not the time to go that fast - I mention this because the locking up you're experiencing may also be coming from trying to push the speed too much.
2. Can you play what you're playing slow? I have had a lot of students that will sweep really fast, but are unable to slow things down, so it becomes more of a party trick, verses an Arpeggio that you can control and bend to your will
I'd advise taking what you're doing at high bpm's and see (as an experiment) if you can keep a hold of it at a slow bpm. This will also allow you time to see/feel what is unhelpful in your technique, and you'll find that things will start to fix themselves, because you'll be able to hear the weaknesses as they happen. Control is 1000% more important than speed with sweeps
Before I write a book - I'll go back to Ed's point because it's so important - Sweeping takes a long time to really have control of, so if you're 2 weeks in, enjoy the process of tightening things up, but you won't be able to rush your own journey - the journey is the fun part!
I'll leave you with three things to chase:
1. Relaxed Sweeps
2. Controlled Sweeps
3. Calibrated Sweeps
If you can tick these boxes, there should be no issues with the speed
Hope this helps!