I am having the same problem with Merlyn, whom I have had since last October. Some days I can put my hand in very slowly to refill his millet seed, other days he freaks out and shakes alot. When I'm not really close, he sings so sweet and now says Pretty Boy. I have been around budgies most of my life and never had one so scared of me. He doesn't like to come out of his cage as we have a lovebird who bites Merlyn's feet.
I talk softly to him every day, sit near by for a while. I wish I had an answer for both of us. I've never tried clicker training, will look it up. Try sitting in a chair, not too close at first but talk to Nana, softly. Each day or two move a little closer see if that helps. I'll try it with mine too. Good luck.