Thats great that your cat gets fresh air & loves to be outside.
Just so you know your cat is safe - either get an enclosure that you know he can't get out of or keep him inside while your gone. If you keep him inside, make a special area for him (if you haven't already), even if it is just a corner. Give him some soft stuff like pillows, blankets, toys...fun stuff he'd be interested in.
You don't have to feel guilty about leaving him inside as long as you know he has activities. When you do want him to go outside, you could both spend time in the garden & make it a special thing between you two. Its just not safe for him to be outside alone. That wouldn't be good if anything happened to him while he was outside & you were gone...Either way - its all up to you