Hi jbrickm,
While this might be perfectly normal I do have some suggestions from first hand experience that I would like to share with you.
Almost a year ago I acquired a four month old cat from the vet that I work for. She was very tame but scared. Scared to the point she shook. But NEVER acted aggressive. So my boss asked if I would consider fostering her. I of course said yes.
Well she spent two weeks in a dog crate with her food, water and litter box in a spare room. I would hand feed her and pet her at least four times a day. Well on week three she finally wanted to come out of the crate and be petted. But it took her another two weeks till she would walk around the room that the dog crate was in. But for those first two weeks any noise and I do mean any she would run into the crate.
Well after she got used to the small extra bedroom I moved her and the dog crate into my larger bedroom. Well I did the above procedure again. But it took only two weeks to get used to my bedroom.
But she was still scared of the other members of the household (human and cat). She was a very sweet and loving to me, but no one else. This went on for a few months.
Well then I had obligations a few states away. So I figured I needed to take her with me. Since no one else could touch her other then me. So I was scared I would come home to a feral cat in my home.
Well I stayed in three different places with this cat over a two month period. So she was forced to see new people and places and such. She acted timid but still ate and drank and followed me around shortly after arriving at each place. She even explored these people houses. Which she would never do in my home.
Upon arrival at home. I had a cat within a week that would walk around my home

. She now will even stay in the room with strangers. Plus she will even take treats and pets from the other people in my household.
She still loves me the most. She still trusts me the most, but in general she is much less shy and like most other cats I have met/worked with.
So my two suggestions are simply this: Either confine that cat to a crate down stairs and pet her and feed her there. So she will slowly "associate" you as the "same" person upstairs and down stairs. Or you could try and bring him into a completely new place (if possible) for a few weeks. This might help to have no choice but to "come out" of his shell.
Sorry for the long post. Hope this helps some

. If you need any clarification let me know.