What are you feeding him? He should still be with his mother at this age. Is it possible to return the kitten to mom and pick it up in 4 - 8 weeks?
You need to help him poop if he isn't going by himself. Rub his belly gently in motion towards his tail. You can use a warm damp cloth to imitate a mom cat cleaning. Then gently tap above his anus until he goes.
Stop giving him milk. Cats are lactose intolerant and the milk will make him sick. You can buy kitten formula from a vet or pet store.
You can also try wet canned cat food - the pate style is easiest for a kitten to eat.
You can also moisten dry kitten kibble with water or kitten formula to make it easy to eat.
If he is constipated you can add a few drops of olive oil to his wet food to soften his poop. A five-week old kitten can poop on his own, though a tummy rub may help if he's constipated.