How long has it been since he stopped eating normally? Has he been having regular BMs? If I was in your situation (which I was), I would call the vet. However, when I called my vet in a similar situation, he advised me to wait it out since she was still eating SOMETHING (just not as much as she should've been). By the end of that same night, I ended up taking Bailey to the emergency vet because I couldn't stand it anymore.
So yeah, I'd give your vet a call. It can't hurt to at least talk to him on the phone and get his opinion/advice. And since I know that you ARE like me when it comes to your baby, it's going to eat at you if you don't call and his eating habits don't return to normal.
Also, you might want to pick up some different flavors of meat baby food (make sure the ingredients are JUST meat, NO garlic, NO onion, NO salt), just in case his appetite decreases even more. I think baby food could've saved Bailey's life, so it's a good thing to have on hand. Even mixing it with her regular canned food seems to make it more appetizing when she's being finicky.
Please keep us updated!!!