When I wear a loose shirt, I button up the front and do a Hack snatch where you reach around with your off side hand, lift the shirt as high as is possible and then draw like normal. I found that with practice, it is only nominally slower than sweeping with a good vest but is much more reliable than sweeping a loose garment. YMMV of course.
Kevin