Hi,
Cats don't do 'spite' they just dont know what it is!
In my opinion I would say that there are not enough litterboxes for the 4 cats. Maybe they dont want to use the litter box where it smells of another cat? Cats are very territorial and so maybe they simply dont like sharing litterboxes...