I think people get jaded by extreme quirkiness every week, and if that's all the show has to offer, then people are going to get bored. The first week's plot lines were pretty weak, but that didn't matter because the show was so different it was just that strangeness that held the viewers. But they're going to have to come up with some good plots and character development to hold viewers from week to week.
It reminds me a little bit of 3rd Rock from the Sun, which started out really off the wall like Daisies, but they had some decent writing, along with story lines and character development, and so the show morphed into a pretty standard sitcom, albeit with a quirky premise, held the audience and turned into a hit. It was the stories and the outlandishly excellent acting, not the quirky premise, that was responsible.