I am a firm believer that known, identifiable, named by products are excellent. Sure I wouldn't eat them and think they're gross, but you're right - the cats like them and they are good for them. The problem I have is that they aren't usually named (i.e. "chicken gizzards" or "chicken hearts" or "beef kidney", etc). All we see on the label is "chicken by products", or worse yet - "meat by products".
The other problem I have with them is the fact that real, identifiable by products make up a fairly small percentage of a prey animal as a whole. When I buy a can or bag of food that starts off the ingredient list with "chicken by products", that tells me my cat will get a meal that is maybe 75% "by products", whereas if they killed and ate a "whole" animal, how much of the meal would be comprised of by products? Maybe 10-20% at the most? I want them getting something that is as close as possible to "natural". A meal of 75% guts, feet, and unidentified parts is not natural.
Even the chicken gizzards, hearts, liver, and the bones are on the shelf at Walmart...if they can sell them for human consumption, they will. These "good" by products aren't going into the cat food. What's left or of substanard quality goes in the cat food.