This is a tough situation. One the one hand, hopefully, you'd like to be able to help the mom and kittens. On the other hand, someone has been irresponsible and now they're in you own cat's yard!
I'd talk to the neighbor's and see of anyone is familiar with them. They may be being fed by someone near. You'd have to decide how to handle them in your yard at that point.
You could also call local rescue groups and ask if there is anyone who can trap them for you. Keep in mind that if this happens, they will most likely be taken to the shelter and, unfortunately, will probably be euthanized.
Is there a way your cat can go outside without going into the yard? A porch or other place he can be confined? I know it isn't fair but he's better off not being able to roam. Too much trouble waits for cats outside - the mommy and two kittens being perfect examples.