We've had a cat door for something like 13 years. In all that time, there has been exactly one instance of another animal coming in -- at least I assume that's how he came in, because we were at work, and no other door was open. That was a neighbourhood cat, who has been canvassing our house for a while, and been told several times by Fawn to take a hike. On that occasion, he presumably went back out the same way, but he left his collar and a good deal of fur behind. It was several weeks before he came calling again. Other critters have ventured in an open door, where they can see what they are getting into, but never come through the cat door.
I think an animal who is unfamiliar with the house, is generally unlikely to use the cat door on spec, simply because he doesn't know what is on the other side of it, or whether he'll have an opportunity to escape once he's committed himself. Our visitor, while he's never been made welcome, has never been mistreated here, has ventured in an open door more than once, and has been around enough perhaps to decide that the risk was worth taking.
Bottom line: I consider the risk vanishingly small, and well worth the convenience for both humans and cats of the household.