There are only a few diseases that can be transmitted by cats. Toxoplasmosis can be caused by handling used cat litter (it happens more frequently from eating undercooked meat, but there is still a tiny risk from cat feces.) If a pregnant woman becomes infected with Toxoplasmosis, her fetus may become infected also. Intrauterine exposure to Toxoplasmosis can cause mental retardation, cerebral palsy, and blindness. I remember a child that I saw when I was in nursing school, in the pediatric ICU. His mother had been exposed to Toxoplasmosis during pregnancy (not sure if it was cats or raw meat exposure.) The child had severe microcephaly, profound mental retardation, was blind and deaf, and suffered respiratory problems (requiring a tracheotomy and ventilator) and had a seizure disorder.
Probably the most common disease that cats can transmit is 'cat scratch disease.' Cat scratch disease is a bacterial infection (Bartonella henselae) that typically causes swelling of the lymph nodes and a low grade fever. It usually results from the scratch, lick, or bite of a cat.
Speaking of cat bites...
Nearly 80% of cat bites become infected. The most common microorganism cultured from an infected cat bit is Pasteurella multocida .
I was bitten by my parents' cat while giving him subQ fluids. I immediately washed out the three puncture wounds and went to the Urgent Care. They washed the wounds again and gave me a script for Augmentin (which I filled and started right away.) The next morning, 12 hours or less, my entire lower arm was red and draining purulent green goo. I wound up in the hospital, on IV antibiotics for 4 days. I missed nearly two weeks of work...and I did all the right things!!! Obviously, cat bites are not something to be casual about!
These are all pretty much "worst case scenarios", but cats really don't cause much disease in humans. I've certainly never heard of cats causing sterility in men.