The absolute minimum is 8 weeks but ideally a kitten should stay with its mom until he's at least 12 weeks. Kittens need much more from their moms than just milk, contrary to popular belief, so the longer the kitten stays with mom the better.
1. Kittens will nurse up to 8 weeks old.
2. They really are not reliable with the litter pan under 12 weeks old.
3. They need more social time with mom/siblings to be mentally and physically ready to leave.
4. Many kitten problems are due to kittens being taken too soon. Most of these social and some physical problems can be avoided by leaving them with mom longer.
Reputable breeders keep their kittens for 3-4 months, give all shots and spay/neuter before you take them home. Kittens adjust quicker in a home at 3 or 4 months old too.
Charlie was born at home, I saw him growing up. Charlie's mom was very kind and we were very close, but Charlie was really aloof. Everytime I touched him, he would ran away, I don't know why. I didn't harm him, I didn't do anything bad. But I'm thankful that he trust me now!
Cats are so unpredictable - we've had Ling since the day she was born (last of the barn cat litters) and her and the siblings were brought into the house at 5-6 weeks old (we handled them in the barn every day).
Miss Ling is not an overly friendly cat and she decides when you can pick her up and how long you can hold her. If she ever got away from us (outside), we'd never catch her as she doesn't even come to you in the house! And she's not one to be bribed by food either!
In my experience it varies a lot. We've had kittens apart from their mother as early as 8 weeks and clinging to them as late as almost 5 months. Generally I'd just let the mom and kitten let you know. Playing it safe, I'd say 10 weeks minimum.