Relatively new adopt has started scratching me while being petted

We adopted a rescue this Spring - 3 year old male tabby. Initially he hid in the basement for a couple weeks. Eventually he cozied up to Amy, and eventually cozied up to me. Was acting super affectionate - wanting to sleep on the bed with us. Apparently relishing being held an snuggled. Liked his belly rubbed, chin scratched etc. Then perhaps 2 weeks ago, his behavior changed somewhat. He still acted like he liked being petted in various ways, then after a minute or two which smack and scratch my hand without warning. I'd go and put out wet food for him in the morning and as I'd walk away he'd take a swipe at my feet and scratch me. Can't figure if he's sick, that this is just is his natural temperament and he just feels secure enough to express it, or he's wanting to play or establish pecking order. The other two cats -- well, one's moved outside permanenty to avoid him and the other hisses at him if he wanders into her territory. I'm in a quandary.

might have to have a vet check him out just to be on the safe side. 

I agree with roguethecat. Cats will get cranky if they have a health issue and are in pain or uncomfortable. He might be swiping at you to communicate that he needs help.
Hope that helps!