That depends on your prescription. As long as you can see clearly without glasses, it diesn't matter if you don't wear them for one day. I have a low prescription and I only wear them when I need them, like watching TV, using computer. It doesn't affect my vision at all.
It should depend on how strong your prescription is and how you feel when not wearing glasses. If you are slightly nearsighted and can see fine without glasses, you don't need to wear them all the time and only need wear them for extreme concentration.