What your doctor said is right. If you can't see things clearly without glasses, you will make the muscles around the eyes tight so that your eyes are easy to get tired. If your eyes often get tired, your vision will get worse. So you'd better wear your glasses all the time to prevent your vision getting even worse.
It is ture. You should listen to what your doctor adviced you. When I was 13 years old, my vison got -1.00, then my doctor advices me to wear my glasses all the time, and I didn't listen to him, then when I go back to have the eye exam again 1 years later, my vision got -2.25. That really surprised me! Wear glasses all the time is good for your vision.