I would recommend spaying and neutering around 10-12 weeks old. While it may be "rare", I have had a few female kittens come in heat as early as 4 months.
I also had a 6 month old male become sexually mature, and bred a litter of kittens. It was a pure breeding, but still an oops.
I do believe that pediatric spaying and neutering will start becoming more widespread. There are becoming more and more vets in my area who are becoming more familiar with the protocols for anestesia and such. There is also more and more research being offered that shows the benefits for early altering, proving many of the old theories wrong.
If your vet doesn't feel comfortable though, I wouldn't press it, he may not feel comfortable with early alter, and that should be his decision. I would try to get it done at the earliest that he will perform a spay or neuter. There is a lot of accredited research on pediatric spay and neutering, that you can find by googling. My favorite articles being with The Winn Feline Foundation and Dr. Susan Little.
My vet will perform at 2 lbs and 8 weeks, but we typically do it at 10-11 weeks old, and they go home at 12 weeks old. They are also 3+ pounds by then.
I do also believe that certain anesthetics can cause problems in young cats, I would get your vets opinion as to what he is comfortable using, and if he has ever had problems using it with young kittens.