I think it is normal. I wear glasses, too. And every time I take them off, I will feel a little pain in my eyes. I guess that is because after I take them for a while, my eyes will go through something different, which may get your eyes hurt but still need time to adjust. And another reason is that you may be draining your eyes. If you watch too much TV or just sit in front of the computer for a long time, you will also get your eyes hurt after you take off your glasses. In fact, this not happens when you take off your glasses, it happens when you put on your glasses as well. It is just a matter of adaption. In order to release this situation, you should do some eye exercise once in while. And by the way, get your eyes checked once a year, and then the doctor will tell you whether you need to change your glasses. If you really hate to wear glasses, why not use contact lens to replace the glasses? I wish this could help you.
Generally speaking, we would be comfortable when taking off our glasses, however, some people rely totally on such eyewear, and thus developed some kind of dependency on them. So there would be some trouble when they take off their glasses. Also, it is likely that you have got something like photophobia, which would make your eyes hurt outdoors.
Your eyes hurt may have no relationship with the fact of wearing eyeglasses. Your eyes may get infection because of invisible bacterium. You need to use the eye drops with anti-inflammation to make the eyes feel comfortable and moisture. You could also do the warm compress to make the eyes feel better.
Because your focus are gathered on eyeglasses lens and the nerves have already been used to focus on eyeglasses. In addition, your fatigue and tiredness are also possible to lead to pain in eyes. If you are still worry about it, then hold a consult with an oculist to solute your problems.