I agree 100%. Your cat will recieve better care if both you and and your cat are more comfortable with one person over another...much in the same way that it's important that you feel comfortable and confident with your own doctor because you will have better, more open communication with that person, which will only only lead to better care for your cat. I had a similar situation: was seeing one vet but then saw a different one on an emergency basis and just loved how he handled and cared for my girl and how he talked to me. From that point on we have only been to see the new vet. Her previous vet was good but we (my cat and I) just clicked better with the other one. There were no issues that I'm aware of over making the switch...I've even seen the previous vet in passing and it was not at all awkward. I think they understand that people have preferences for whatever reason and that ultimately, it's the care of the animal that matters.
