I agree, he's a bit too young to be away from mum, Siamese are a very 'emotionally dependent' breed and need a bit longer before getting their independence if that makes sense? Most of the breeders I know monitor their Siamese very carefully to judge when they are ready to be adopted, and in some cases have kept them with mum for up to 18 weeks for particularly sensitive individuals!
They can also be miserable if left alone for any length of time, if he's the only kitten in the home and you go out to work, I suggest that you consider getting another kitten (my Oriental's breeder would not have allowed us to adopt a single kitten if we hadn't already had another kitten at home!) - it doesn't have to be another Siamese, you could have a look at shelters to adopt in the new year.
As far as the food goes, he's just had a huge emotional upheaval and probably doesn't have much of an appetite, but personally I think a good quality wet food is better for them in the long-term, so if he's keen to eat wet food then offer him that. I'd rather have a cat that refused dry but ate wet than the other way round!
Congratulations on your new friend, Siamese are wonderful creatures and I know that you will have a very special bond with that little guy