Do you have any music/radio on in the room your budgie is currently in? If he's stressed silence is just going to make him more stressed.
I'd keep the solid door shut whether you are in or not. It will make him feel safer for certain. And it might help keep your cat away because he won't be able to see the bird. He'll know he's still there but seeing the movement will draw more attention.