Quite a bit of theory involved but;
To work out what key a piece is in, you would look examine the chords being used. For example, if your progression was: C | F | G | Am – I would recognise the scale needed as C Ionian – or C major pentatonic. A minor pentatonic would work also. You are looking for a key where you could find all these chords from. If anyone could correct me on this but I believe D minor pentatonic would work also.
For some heavy theory;
If you had for example: Dm | G7 | Dbm | Gb7 | Cb |, you have two groups of notes coming from different parent scales. The Dm | G7 comes from C major and the Dbm | Gb7 | Cb comes from Cb major.
Now if we remember our modes, in C major we have;
C Ionian (major
D Dorian
E Phyrgian
F Lydian
G Mixolydian
A Aeolian
B Locrian
So looking at our Dm – G7 I would use D dorian (or minor pentatonic), and we would use Db dorian (or Db minor pentatonic) for Dbm – Gb7 – Cb.
To be fair, alot of backing tracks let you know what scale you can use over the progression being played. But in a nutshell, its working out where the chords come from and how they link together. Sometimes there might be more than one way.