I agree, the main thing to do is give it time. Do you have any idea about where he was before? If he was a stray that just wandered along and picked you, then lord knows what he had to deal with before while living outside. I honestly don't think most of us would survive the way stray cats do. In time, he will feel more comfortable and settle down. Does he have a safe place to go to when he feels stressed? IE a separate room, corner, closet, etc? At this point, he probably needs somewhere where he feels completely safe and no one should bother him.
Also, I would recommend feliway.
http://www.feliway.com/us
Basically I agree it takes time and patience. Bless you for taking in the sweetheart and giving it a good home and a chance at life.