Since you don't know this guy well you have no idea what kind of creepy phyco he may be. I would politely go to his place of work, pull him aside, and tell him that going to your house is unappropriate. Let him know that if he chooses to do it again, you'll call the police. That way he's forwarned. If he's a normal guy he'll stay away. If he's the creepy freak he seems to be he'll show up again and hopefully the police will take care of him. If I were you just to be on the safe side I'd be careful to always lock the doors and windows, especially at night. It's better to be safe than sorry.
