There are a lot of sites that do this trick. The cards do change. If you write down all the cards and then doublecheck them in the 2nd viewing (which flashes so it's hard to do this) you'll find it's true. I just remembered that the Kings were hearts and spades in the first grouping and the Queens and Jacks were hearts and spades in the second grouping.
Yep, that's the way these things work. There's always a mathematical reason or method hidden in there, so always strip it down to logical thinking. How could it be done and always be right? In this case, the only way for him to make it look like your card was taken is to take all of them away and replace them each with something similar. Like was said, the trick counts on the probability that you will only pay attention to the card you picked.