Here's my advice for what it's worth...
Call your local no-kill rescue groups, tell them the story, and see if they can help. They do want to keep these things from happening, and especially if you're willing to hold onto the cat for even a little while until they get it a foster home, they might be able to help you, and in a way that might not affect you at work.
case in point...When Deja Vu had her kittens, and I contacted the rescue group, I had already promised one of the kittens to my neighbor. She had never had a cat before, but wanted one badly, and after I explained what it needed, the responsibility involved, how to take care of and raise it, and the special needs it had as a kitten, she swore up and down she understood and would take care of it right and love it.
First night she had it she gave it food, water, and went out. Left that poor baby, who had just been ripped from it's Mom and 4 siblings and thrown in a strange house, completely alone to go play cards with her friends. After I told her not to leave it alone (except for important things like Drs appts,. etc) at least for a few days and she swore she understood and would not.
She called me when she got home to tell me the kitten was hiding under the bed and wouldn't come out, and could I help. I went over, and as soon as she saw me she ran up to me and into my arms, and shook and howled like I'd never heard for about 1/2 an hour until she calmed down.
I didn't know what to do. I was so tempted to just walk out with her, but like you, when you're in a situation that's difficult, and I have to live next door to this idiot, you sort of freeze sometimes and I left there making her promise if she had to go out the next day she'd leave her with me until she returned.
The next day I realized I did the wrong thing and should have taken it. I called the rescue agency, and together we came up with a solution (okay, yes, we lied, but I think the means was forgiven for the soliution).
I told her that the rescue group was coming the next day to take the kittens to get their shots, etc. and put them up for adoption, (up to this point that was true) and they offered to do the same for Rocky and the other kitten I kept (which was with her) for what it cost them, then bring them back, which would have saved us both lots of money. Since they were coming at 7am it made sense for her to leave them with me that night so they could take them both.
Once I had her back, and the group took the kittens, I apologized to the woman and told her I had made a big mistake, and since the rescue had invested some money in the kittens (shots and a vet checkup early on) I hadn't realized it wasn't mine to give, and she'd have to apply to the rescue group to adopt her. I apologized profusely and said I hadn't realized that. The woman from the rescue backed me up. When my neighbor started to apply, and found out how much it would cost her to get the kitten through the group, and all the other things involved, she decided it wasn't for her.
And nobody's feelings were hurt, no anger, and the kitten went to a fabulous home (the daughter of a vet).
Anyway, the rescue group might have a solution for you, or help you with what you can do, and at the very least, eventually put the cat in another foster home if it is hard for you guys to keep him.
Can't hurt to call. Good luck.