Cats do need to be protected against rabies and distemper. The question is how to do this best and most safely.
The general thought now is that one modified live or two killed distemper vaccines, given after 6 months of age, give strong lifelong protection. This vaccine is pretty benign, though, so you might want to go ahead and give it every 3 years or so. I do not think annual vaccination is necessary under any circumstances.
As for rabies, if he has been vaccinated once, you can ask the vet to run a protective titre test instead of routinely vaccinating. That way you know if your cat is protected against rabies and you don't have to vaccinate him unnecessarily. This is important because the rabies vaccine has been associated with vaccine sarcoma.
As for the others, it varies. The FeLV vaccine is protective but also is strongly implicated in vaccine sarcomas. The virus requires direct body fluid-to-body fluid contact for transmission (and even then only about 30% of exposed cats will get the virus) so the odds of your cat actually being exposed are very low, especially if he is supervised when he goes outside. This is really a case where you have to weigh options and make the decision that you feel is best. The vaccine is not risk free but leaving your cat unvaccinated is not risk free if he goes outdoors. If you do elect to go ahead with the vaccine, I would definitely recommend asking the vet to give the shot in the cat's tail. The reason for this is because if a tumor does develop, the tail can be easily amputated which will hopefully stop the cancer in its tracks.
I haven't heard anything about the FIP vaccine actually being harmful but there is also not much evidence that it is helpful. 75% or so of all cats carry feline coronavirus, which is the virus that causes FIP. However, straight coronavirus isn't what causes FIP - it's caused by a random mutation of the virus. No one knows exactly what causes the mutation but the vaccine will not keep your cat from developing the mutation if he has feline coronavirus. And even if he does not have coronavirus now but gets it someday, still only about 5% of cats who have coronavirus ever get FIP. So even if the vaccine is low risk, I just don't have much faith in a vaccine against a disease that is itself so poorly understood.
The FIV vaccine is one to stay far, far away from. For one thing, FIV is extremely difficult to transmit - it requires an infected cat's saliva or blood to enter your cat's bloodstream. This is common in unneutered tomcats who fight but it is very uncommon in other circumstances. But more importantly, medical technology currently has absolutely no way to tell the difference between a cat who has FIV and one who is vaccinated for FIV. The only tests available test for the presence of antibodies, not the virus itself. So a cat who is vaccinated for FIV will test positive for FIV! This means that your cat's life would be in jeopardy if he ever ended up in a shelter, plus if he did happen to get FIV (unlikely, but of course not impossible), you would have no way of knowing. It's just not worth it.