It does sound as though he is scared if it is only happening when you are taking him out of his comfort zone. Are you physically picking him up or catching him? Budgies do not like to be enclosed in your hand and feel threatened when this happens. Is his cage covered on three sides? This helps them feel secure and safe as well.