I volunteer at a shelter, and I always make sure I wash my hands and halfway up my arms before I leave.
I mean to change my clothes as soon as I get home, but I often forget if I go somewhere straight from the shelter, and do a few things before I come home. Our shelter tests all cats for diseases, and they're in quarantine for at least a week before I get to see them, so I'm not too worried.
If a cat has something like tapeworm, a flea that's eaten a tapeworm egg would have to jump onto me, come home with me, land on my kitty, and have her eat it before she got it, and she's been "Advantaged", so the odds of her getting something like tapeworm even if a flea did manage to come home with me are low.
As for kitty cold, I'm not really sure how it gets passed from one cat to another, and at what stage they're infectious. If any of the shelter cats has runny eyes, runny nose or so much as sneezes once, they are checked over. Because there's volunteers with the kitties for 6-7 hours a day, kitty colds get picked up pretty fast and they go straight into quarantine. I'm confident my shelter is clean enough for me not to bring anything major home to my kitty. We're only a small shelter, so they're pretty easy to monitor.
It may happen, but I'm not worried enough that I feel I have to strip off the moment I walk through the door and take an immediate shower.
Assess the situation when you get there, and ask the vets there what the risk is. I'm sure they'll tell you what they recommend.