My indoor/outdoor cat is depressed after being confined in my own yard.

We moved to a new house with my 2 cats and 1 dog about 2 weeks ago. My cats are used to be indoors and outdoors. Speedy is a girl, 6.5 years old, Spotie is a boy, 4 yr.s old. They both were rescued from streets when Speedy was 6 weeks old and /Spotie was 6 months old. They were spayed and neutered. Spotie is the problem. Spotie used to roam freely in the neighbourhood. There were many trees and grass, birds and lizards, as well as cars. Speedy will go out but always close by home. Spot, we don't, have any idea how far he used to go.
In order to keep them safe, we installed cat proof fence in the new yard. Spotie escaped once. And we fixed that particular corner which closed to the entrance gate. Spotie is not happy past few days, he walked around the yard and looked up to the fence constantly to try to find the escape route. He was upset when he couldn't find any way to escape. I encourage him to play with toys. But he is not even interested. He looks depressed. Tail is always down. We just moved in and the house is full of boxes. The yard is not big and there is no grass nor tree, only has some plants. We do plan to plant grass and make it more appealing. But right now my boy is not happy. He sometimes hides under the kitchen cupboard and put his face towards the wall. Am I do the right thing to keep him in my own yard? Can any one give me some advice?

The idea to install the cat proof fence was to keep them in my own yard. We don't want them to be in any danger outside.

Can you plant a tree in the yard?  Build some kind of kitty fort for him to climb?  Looked into outdoor cat trees/furniture?  Install a bird feeder high above for birds to visit (but of reach of the cats)?

Thanks I will look into to build one for him. But it would not happen until we fully settle down.

Spotie hid under the kitchen cupboard and faced the wall!