The golden rule with litter boxes is to have one per cat, plus one. So you should have two boxes, and judging by kitties behaviour I'd say the other one needs to go where he's pooping now! Some cats don't like to pee and poop in the same box, they want seperate ones for each function. Maybe there's something around the first box that he doesn't like - loud noise, washing machine, too exposed? Cats like to feel safe when they use the litterbox as in nature they are most vulnerable at this time, make sure that when he uses the box he can get a good view around him, but also it's private enough for him to feel secure. Make sure the box isn't tucked away in a dark corner where he may forget where it is!
Is his litterbox kept clean? Cats are naturally clean creatures and he may poop outside the box as a way of telling you it's not clean enough! Check the litter you're using too, he may not like it.
If you think he may be lonely, have you considered a playmate for him? The devil makes work for idle paws, and two kittens will keep each other out of mischief as well as providing company for each other when you're not there!
Good luck and good on you for adopting a stray baby!