I have read a little about the history of underwear. The history of men's underwear is very varied and full of chapters. Now, when it comes to women's underwear its less full. Panties as we know it was a recent ocurrence. During Roman and Greek times women would often wear a tight breast band, serving as the precursor to the bra. However, for underpants, they weren't wearing anything. This was the norm all the way into the Rennaisance. During the Rennaisance and the Enlightenment only women who were actors in theater would wear underpants. (If you are doing a play on stage, you can't just go to pee in mid event) In other words, Martha Washington was wearing nothing behind that petticoat. Corsets would do the supporting of the bra. Nuns were in the same situation. I doubt though that a nun would wear a corset during the Victorian period, as their habits were so simple.
During the Victorian period, panties started slwoly getting into vogue. It was a loose fitting pant of sorts worn under the dresses. And not all women would wear it. A French woman who was tired of corsets invented the modern "brassiere" during this period.
By the twentieth century we have modern female underwear, Hanes, Playtex and all the multinational brands, and the cotton briefs everyone's so thankful for.
So, to answer your question, of what did nuns wear under their habits back then: Nothing. Absolutely nothing.
Now, what I never got around, and nobody has been able to tell me was... how did women deal with their periods then?