Well, they don't understand how the phone works, so they don't know what you're doing. The cat either thinks you're talking to him (but in a very different way than you normally do), or talking to yourself. Either way, it can be unsettling.
My cats go to the answering machine when someone leaves a message. If I get home and my answering machine is not on the table, there is usually a message, so I assume my cats get more aggressive with the disembodied voice on the answering machine when I'm not there.
I sometimes wonder if the fact that my spouse usually meows to leave a message has anything to do with it.
But my cats (other than having to be dissuaded from attacking the mouthpiece on my headset phone) are totally comfortable with me talking on the phone. In fact, it's much easier for me to trap Artemis in the bathroom for drug dosing when I'm talking on the phone (I figure she figures I'm distracted).
When I was on vacation and my dad was home watching our cats, I called my Dad and one of our cats who is very close to me perked up when she heard me over the phone.
Always. Inevitably Sho's butt ends up in my face...and Tomas starts trying to get into stuff. So anyone I'm talking to usually hears "NO! GET DOWN" "GET OUT OF THAT!" "TOMAS!" "you're not getting any food..."
or the favorite.. "get your butt out of my face" and "oof" from Sho walking on me.
When I'm on the phone, Daphne has to be right there, draped over my shoulder. That would be okay, but she also wants to dislodge the phone while purring loudly