It's normal but not necessarily okay. This is a good time to switch from clay-based litter to wood, corn, wheat, or paper-based litter. These things won't harm the kittens if they nibble a little.
Most often, kittens ~will~ sample whatever type of litter you choose but I like the regular, unscented clay type the best for kittens. If they do eat a little, most of the time, they do not swallow it, they spit it out. But with other types such as the newspaper, wood or other, it degrades when wet and is easier to swallow. I would just stick with the regular clay kind.