Yeah get those kitties and as someone else said "fair game". I am the ultimate conflict-avoidant person and my avoidance cost my foster boy Curly coming down with FIV. I was told he was the neighbors cat and yeah he was, but abandoned.
Every single time I have ignored my instinct I have had to do much heavier lifting than if I had acted promptly right from the beginning. So trust that if you are doing a lot of feeding, sadly the people on the other end aren't caring properly for the cats.
If you do have access to spay/neuter low cost, you can always also circulate a flyer offering your services and strike up a conversation that way, but still take the kitties as they appear and TNR them, feed them, treat them and put them up for adoption if you have access to a group or are part of one. The most important way to find out if you have a caring family on the other end, take the cat, wait a week, call all the hospitals police etc and see if the family reports it. My families never have said anything.
I have learned over many years and the hard way that always it is better to just take the cats and wander back later, innocently, with inquiries if you so choose. So long as you do the vetting etc. you will never get yelled at since it saves them money.
No matter what you can always go back if you choose, but you can also just move the kitties to a better place if your gut tells you that is where they belong. Either way you cannot go wrong doing what the cats need and later approaching the neighbors.