Despite what vets were taught, maybe many years ago, neutering does not affect the cat's eventual size. Male hormones do change the shape of their face; neutering early might not trigger the broad cheeks and thicker neck of the typical tom.
PHYSICAL EFFECTS OF EARLY NEUTERING
A neutered kitten's growth is not stunted, although males do not develop the characteristic tomcat "jowls". In a parallel scheme for dogs, early-neutered puppies were found to become "leggier" adults than those neutered at the more usual age as the sex hormones appear to affect the 'switching off' of bone growth. Though no data has been collected, the same may be true of early-neutered kittens. Fears about stunted growth are unfounded. Legginess and the absence of tomcat jowls is only of concern to exhibitors (breed standards should make provision for neutered cats having a slightly different appearance).
It is hard to assess whether earlier neutering has any additional positive effects on behaviour (other than preventing sex-related behaviours), but no adverse behavioural effects have been noted.
http://www.messybeast.com/earlyneuter.htm
On a personal note, I adopted a Maine Coon mix from a shelter two months ago. He was 4 1/2 months old at the time, and his "package" was really prominent for that age. Maine Coons take a long time to mature, yet kittens under adverse circumstances can go into puberty early, as this is nature's way of trying to get a new generation going before it's too late.
On balance, I decided it would be best to get him neutered before he reached 7-8 months, which was my usual vet's advice. The quote above was part of my research. So once he had fattened up and had no health issues, I found a vet who understood my rationale and found no medical reason to disagree with me. I was very concerned that he would start spraying, and once they start, neutering does not always solve the problem.
I did find that the litter box became stinkier, which I attributed to his metabolism healing, because he'd gone without food for so long in his past that he lost his down (the soft fluffy underlayer.) We feed canned Fancy Feast & Nine Lives, and Evo dry (no grains.)
I fixed the litter box stinky problem (baking soda, my routine additive, wasn't cutting it) by adding Cedariffic litter to my clumping litter. It still clumps and the Litter Robot still works, the smell is gone! You might try that.
BTW, the breeder probably has intact male cats on the premises, thus the odor.