It is better to leave them out for at least 24 hours before the exam. The doctor need to check the real condition of your eyes. It is the same as you should not wear glasses when you are in an optometry.
You should take your contacts out a day or two before your annual exam so that they can test the real condition of your eyes. The exam will detect whether you have got any eye problems or whether your vision has become worse or not.