WHEN was he wormed by a Vet? Was he having these issues when he was last seen by your Vet? If so, what did the Vet say about them? Anytime a cat is losing weight and not on purpose, they need to be seen, and particularly if there is also a change in their stools. I would go sooner than later.
Gray coloured stool would indicate that he isn't completely digesting his food, and the accompanying weight loss suggests his gut isn't processing/absorbing nutrients 'properly' - getting sufficient nutrition.
You really need to take him to a competent feline Vet - the sooner, the better.