I'm so sorry to hear about your struggles with Harvey. I'm sorry that I can't recall his history, but I do know you inherited him from the woman whose home you moved to, but don't recall how long you've had him.
Mental illness in a cat is very difficult, if not impossible to diagnose. Harvey may have been a victim of abuse previously, or he may have some wiring issues in his brain. After 7 years of struggle with my baby Oscar, I finally released him from his mental illness on Monday. I apologize if a ramble a bit in this response - this is still emotionally raw in my brain.
Recognizing the difference between an aggressive cat versus one with a physically based mental illness is not easy. If Harvey's behavior is based on past abuse, then you do have a real chance to work with him. Have you read the sticky thread about cat aggression towards people? If not, start there.
I never found a vet who had any ideas on how to diagnose mental illness. The closest I got was the admission from one of my vets that had a bi-polar sister, and the behavior I described to her about Oscar was very similar to that of her sister. I also have a bi-polar brother and I recognized my brothers symptoms in Oscar. As the vet said, what happens in humans can happen in cats and a great deal of their behavior is genetically based. You won't get much help from a vet other than prescribing medication to help even out Harvey's moods. The question on whether to euthanize due to possible mental illness is entirely based on quality of life.
The things that I noted about Oscar that confirmed my suspicions that something was fundamentally wrong with him were as follows. Perhaps you'll see something in this list that you can apply to Harvey.
- The inability to relate to other cats on a consistent basis. There were times that he snuggled up to them, did mutual grooming and was very calm with them. Other times the other cats would try to dominate him. Other times he would simply hide under the bed to avoid all contact with them. The level of his avoidance increased tremendously over time.
- The inability to do things that cats normally do. Oscar was terrified of toys. Whenever I pulled one out to play with him, he would immediately hide under a bed. He might lay still and watch other cats play with toys, but he never once attempted to be part of that play. By watching other cats play, he should have gained an understanding of what toys were for, but he never got it. He did the same thing with treats. I'd toss him a treat and he'd hide from it.
- A complete inconsistency with his interactions with humans. He was good with me for years and usually hid from strangers. Then he would surprise everyone by walking into the room with a complete stranger and rub against them for loving. Over time, he hid from me more and more.
- Unexplained changes in mood from moment to moment with no apparent trigger. Now many cats have extreme mood swings and this one was more difficult to judge. Oscar used to like to jump into my lap and head mash my face. During these times he would purr and give me that look of total adoration. I would have my arm wrapped around him to scratch his face. As he got older, I noticed that his body would go from completely relaxed to frozen in fear in a heart beat. His pupils would also dilate. Then he would leap down, run under the closest bed and sometimes stay there for a week. I really paid close attention to these times, and in particular looked for external stimulation that may have triggered his fear response. There was nothing happening at those moments.
- All of these symptoms got slowly worse over the 7-1/2 years that he lived with me. His "good" times when he was sociable with other cats and me dropped to about 5-10 minutes a week. He was good most of the time when he was younger.
- What finally pushed the situation over the edge for me was when Oscar's behavior went from passive/fearful to aggressive. Because of his fears of other cats, I had him in his own room away from the other cats and he was on anti-anxiety medication. I would spend time with him each day using many of the tricks that you use with feral cats. Sitting on the floor and reading to him, never approaching him directly, no direct eye contact, etc. He actually seemed to be relaxing in this environment. Then one day he started to show signs of aggression to me. It escalated to the point that he went on full attack mode with me - the best I could describe is violent. For a cat with suspected bi-polar illness to switch from a passive/fearful cat to a completely aggressive cat was not normal. I've dealt with feral cats and I've never seen even a feral cat with this behavior.
I brought Oscar to the vet to change out his medications. Because of the fact that his behavior had gotten continuously worse over his life, we knew that his aggression would only escalate. That one quality of life remnant that he still had was gone. It is a horrible thing to try to figure out quality of life from a behavior standpoint.
Back to Harvey. Don't let anyone else influence your opinion on his behavior. You know him better than anyone else. Look for signs of true emotional instability versus a simple response to his environment. Keep a very close eye on things that could be triggering his aggressive behavior. Remember that a lot of things can set off a cat - a change in your routine, a change in your emotions, a new cat in the neighborhood, an illness in your household (are all of your cats healthy right now?). It took me a very long time to come to the realization that Oscar was never going to improve - and I'm not going to second guess myself that I waited too long to reach that conclusion. When I finally checked my emotions and looked at his behavior rationally, I could see what was happening plain as day.

to get your through this. Please PM me if you want to discuss this privately.