Brandon, no one here is saying that you should give up your cat. Laurie does rescue cats and kittens, and our research for www.SaveSamoa.org
and especially www.StrayPetAdvocacy.org
has shown us more than we ever imagined about the immensity of the pet overpopulation problem that plagues the US and the world. Hence the passion that this subject brings up. Honestly, everyone who allows their cat to breed indiscriminately is adding to that problem and adding to the huge number of healthy and adoptable dogs and cats that are killed each year, each day in shelters. While we are sure that you allowing your whole male outside wasn't intended to add to this, it is highly doubtful that he hasn't fathered kittens at some point in his wanderings. That he has returned with injuries confirms this, since toms generally fight over females (while females fight to either protect their kittens or over territory).
At the very least I encourage you to spend some time in our SOS forum, and our Feral Colonies forum to see just a small sampling of what can and does happen when unwanted litters happen.
And that is in addition to the health and behavioural benefits of spaying and neutering.