It can take a while. Since rabbits are prey animals they are often shy around strangers and in new places.
I bonded with my first bun Mocha in maybe a week, Loki a couple of weeks. Fey and Sprite, who had been abused and neglected, took longer to develop a personal bond with because they were so scared. Maybe a month of trying hard? Oberon was near instant- actually I felt a strong bond to him when I first saw his picture. By the second day here he had me eating out of his paw, LOL!
I've found the majority of rabbits are very interactive when you give them a chance. They prefer you to be on their level. Sit or lay on the floor and pretend to ignore them, try reading a book. Definite bunny magnet! If he needs more encouragement, put a few toys and treats around you to encourage him more.
Also, learn about rabbit body language. When I first got Mocha I didn't understand her at all. I had a lot of her body language backwards. For example, I thought that she lowered her head when I went to pet her because she didn't like it and was shrinking back from my hand. WRONG! She was actually demanding I pet her more! This site is great:http://www.muridae.com/rabbits/rabbittalk.html
As for hay allergies- keep the bale in the garage, use an air purifier in your room, and vacuum frequently. Take allergy meds too. I hear you, but I love my buns too much so I just deal with the allergies.