You know you shouldnt look directly at him? Look a little at the side.
IF you look directly, look with eyes half closed, like friendly cats do.
You can also yawn a little. Or even standing on all four, stretch yourself, forwards or backwards. Like cats greeting a pal or his human...
Raised hand is a threat (but homecats are used to this, and also people looking at them = part of being a homecat
But a low hand stretched forwards if friendly.
When you feel he is behind you, try to get lower. Sit down or something before turning around. So he will perhaps dare to stay.
He knows already you are his almost pal. I think he will recognize the signs - and hopefully - also accept them.
Good luck!
