Defecating outside the litter box

I'm hoping to connect with cat owners who have had this experience. I adopted a 4 year old cat almost 2 weeks ago and he has settled into his new home. He is very friendly and affectionate. We have noticed a preference by him as to a location for his litter box and have supplied him with 3 different boxes in 2 locations. The problem we are having with him is that he defficates just outside the box. He did use the box once and I witnessed him using it. He didn't seem to have any issue with pain. One box is low, one is filled a little higher with litter and a third is full 1-2 inches with unscented wheat litter. The other 2 are filled with cat attract brand of litter. He uses all three to pee but defficates outside on the floor near the boxes. We have tried seclusion and he will use the box then but as soon as we let him free he goes back to pooping on the floor. We've used an enzyme spray after cleaning up the spots. Does anyone have advice for me?

I'm wondering whether a continuation of the seclusion may be the way?  Bring him out for play, cuddles etc but put him back into his smaller area the rest of the time?