If your cat is not used to you being away from home for any extended period of time, then yeah, he might very well "flip out" while you're gone. It's really hard to predict. Is he normally a high-strung, clingy cat? Or is he more laid-back?
Shortly before you leave, could you get a T-shirt sweaty and put it under his food bowl, or somewhere else around the apartment where he likes to hang out? This way he'll still have your scent around, which he'd probably appreciate. In his mind it might be the next best thing to you actually being home.
Most important, is there any way that the friend that you mentioned, or somebody else, can stop by your apartment at least once per day, not just one time on the second day? I think that would certainly help matters. The litter box(es) would be cleaner, at the very least. And if your cat eats canned food, they would be able to give that food to him, rather than just replenishing the dry food bowl and changing the water. And you didn't mention it, but these people should also spend some time playing with your cat, if he's willing. He's only 2 years old and probably has too much energy to just sit around and sleep for the three days that you'll be gone. Of course if he doesn't already know these people or is a scaredy cat, this might stress him out more.
I can understand your reasons for not wanting to board your cat. I feel the same way. But one of the benefits of boarding is that there are people around to cater to the cat's needs. It's obviously impractical for you to replicate that arrangement at home, but please try to have people stop in to check up on your cat more frequently than you had intended.