I always handle kittens from the first day. I give them a few hours to get settled but within the first day, I pick each one up and do a quick visual exam (sexing, making sure they have all the external parts they ought to, checking for cleft palates, getting an initial weight, etc). From then on, I handle them for at least a few minutes each day as long as Mom is okay with it.
Mom won't reject them (even if Mom is feral) and it's very important that the kittens are handled frequently from the earliest possible point so they are well socialized with people. When they're about 4 weeks old, you can start more intensive socialization such as inviting school-aged kids from the neighborhood over to play with them.