I'd say worms too. But our cat is the same... she gets two meals of wet food a day, dry food for munchies whenever she wants, and still bugs us all for nibblies!
We've considered changing her food from various tinned to kangaroo, to see if it bulks her up a bit, and fills her more... But when we tried it last time, she went absolutely nuts tearing around the house at full speed.
Some cats just can't seem to get enough food. Our cat is 1 and is still a growing machine.
Something else to look into, is what foods he goes nutty for mostly. For example, my cat at my mothers is a nutball for corn, and my mother and I love our corn. If I turned my back, she'd steal half the corn from my plate before I could blink. If you are having the types of foods he loves, he may be going nutty because he likes the taste, as well as being a growing kitty.
Our cat, Bastet, loves fish of any variety, pasta sauce with or without cheese, tinned asparagus and anything with milk.
So it could be in part that he's growing like a weed, but also in part, you have the same food tastes.
Another thing to try, which a friend of mine did, was to feed the cats in another room before you sat down to eat. I felt it worked well, because a) their evening meal was done before you even sat down to dinner (7pm for most people :P) and b) they're busy with their food and ignoring yours. The trick with that is timing. You have to put their food down before you start serving yours up, and not let them see that.
Good luck! We've sometimes put Bastet in the hallway when fish was served... So I feel for you /hugs