Inbreeding is common in feral colonies. Inbreeding is what causes birth defects and other problems with the kittens. If the colony continues to breed within itself, it progressively gets weaker with more bizarre side effects of inbreeding, such as tumors, and blindness. Polydactyls (cats with 6-7 toes on each paw) are a result of inbreeding of a feral colony. Ernest Hemingway was so taken by the abnormality, he kept his colony together to keep this trait strong.
The best thing to do is trap-neuter and release, that way the colony can stay together but no more breeding comes into play.