Unless you are home all day and supervising your budgie (as in truly watching over him) for the whole day, you could hypothetically do that.
Still, it's not very realistic to assume you would be able to do that and while you are in a different room from your budgie and he happens to come out of the cage while you are unaware, there are risks for your budgie's safety, especially if he is given free reign of the house.
You may come into the room to not find him, he can get into trouble if he accesses a room that is not bird proofed, he could take advantage of a forgotten open window on an area you weren't expecting him to reach to make an escape, etc.
Certain household shores such as the various kitchen duties (prepping and cooking and having meals, doing the dishes on the sink, etc) should be done when the budgie is safely inside the cage with the cage's doors closed.
Other potentially dangerous chores include vacuuming, ironing, washing floors, etc. Anything that keeps your focus away from the pet bird, allowing him/her to get into trouble when having out of cage time is to be avoided.
To sum up, for overall safety reasons, it's simply best to leave the cage's door open only when you are home and able to give Yogi your undivided attention for the duration the door will be open for him to come out if he wishes to do so.
If you start working with him in terms of establishing a bond, building trust to later on begin training (by teaching Yogi the step up to your finger command), this will make the out of cage time less stressful and it will be much easier to take him out and bring him back inside the cage.
You will have all the detailed information by checking the stickies at the Taming and Bonding section of Talk Budgies.