3.5 yrs. with 7 kids. Breaking bad habits

[Buzzy]

We purchased a beautiful Timneh from a family with 7 kids. We knew problems would be there but we accepted them. Diego, our Timneh, has come along wonderfully in the last 3 months living with us. All is normal with one exception. He is afraid of my hands. He will step up from his cage, and will go to my arm, but to try to touch him, your in for a fight, and missing hyde. I was thinking of toweling him & taking him in the bathroom (quiet, no distractions) and caress him, spend time talking to him and generally assure him I'm not going to hurt him by touching him. Latter, towel games will come if he's acclimated to this treatment.
Does anyone agree, or have a different or better way of dealing with this behavior?
Buzzy's got the blues.