What do you find most challenging caring for ferals/strays?

For me, shoveling the entire yard to make a path for them if we get more than a foot of snow, so they can safely get in/out of their outdoor cat houses and not get trapped under the snow, and to make room for their food/water bowls.
With this past Winter being insane I am happy to see the snow gone, and so are the cats.

Also, when it rains and their food gets all soggy and I use up so much dry food replacing the soggy food.. There is no place outside out of the rain that can keep the food dry.

For me hardest is perhaps to avoid eye contact and trying to look small.

Shoveling paths to snow was actually kind of fun, but then again we did not get any wet snow, only that light stuff and it is already melting quite well, still having nearly 2 foot of snow left and up to 6 at piles.

Old garden table top put that leans against house can already provide some shelter from rain, which could save in expenses.

I've been feeding a colony (originally, three to four; now eight to ten) in front of my condo complex for over a year. I've TNR'd about twelve.
The hardest thing has been dealing with the humans, particularly Grumpy Old Man ("GOM"). I can't control cats' behavior: Patches is a digger; four to five cats seems to prefer the GOM's ground floor balcony to shelters with stray in a protected, somewhat heated utility closest; some like to dig in front of his balcony; many like to walk on the roofs of cars. (I gave GOM a can of cat deterrent; I don't know if he used it.) I've let owners/renters know that the "R" in TNR doesn't mean removing, it means returning; they don't understand/believe the vacuum effect.
All I can do and say is: TNR prevents cats from having kittens; and you will continue to see cats because people will continue to be irresponsible cat owners. (Yes, I've posted literature from Alley Rescue Cat and Alley Cat Allies. Didn't help.)
Rant/vent over

Gosh, I have a few. Let's see. Yes, the shoveling of the paths this past winter when we had seven foot drifts was a back-breaker. Trying to figure a way to keep racoons and possums out of my cat's shelter eating all of their food really irritates me. The racoons make a muddy mess of their water too It's like they wash their paws in it. OH, and the complaining neighbors, or should I say the cat hating neighbors. Some are very compassionate about the ferals and others not so.

I really love caring for them all in all and don't mind any extra work I have to put into their care. And, I don't let the neighbors bother me, I have a whole speech to lay on them and my "donation to the cause can" if they come around asking about the cats

Well, we live in a really rural area, and neighbors aren't a problem. There's so much natural shelter, they don't use the ones we put out. We have to shovel for ourselves, not them. There are a lot of ground hog dens around here, and when it snows we worry they're buried in there... but without knowing if they're using any specific one, there's nothing to do about it.

Keeping the heated water dish clean was a PIA. Cleaning a water dish in 0 degree weather is hard on the hands! We had the same problem with the racoons LOL.

But for us, I think the most challenging thing is just worrying about them. We stress far more over them than our indoor babies, worrying about whether or not they're OK - because it's so much harder to care for them if they need attention. We've been lucky in that the ferals that have needed medical help (so far) have come to us and we've been able to get them to the vet... but we worry.

We moved back into the RV in the summer of 2008. Since moving back, 15 cats have gone into the foster network and been adopted out, 13 more have been TNR'd (most of those just this past February/March - the bulk turned up this winter), and we have (at least) 2 that still need to be TNR'd.

Like Laurie, it is stressing over them, particularly if one of them goes missing or when they get old.

I always found places to feed them where their food wouldn't get mushy (under a covered porch or the patio table). I looked at shoveling as a way to exercise so didn't mind that. I worried a lot about cold weather in winter, but finally found a way to build a shelter with a heated bed to keep them warm.

I lost the last member of my feral colony last summer at the age of 18. I no longer live in an area where ferals congregate, so my stress level has gone way down.

I want to add..... I Third what LDG and Momofmany say. I do worry constantly about them, especially if I don't see someone who I usually see everyday, like Yogi. Then I am very concerned. So the worry is stronger for the ferals than my inside kitties who I know are always safe.