Yes. You need to remove contact lenses before using allergy eye drops. Firstly, the contact lenses may block the ey drops from touching some parts of your eyes. Then your eye allergy cannot be treated effectively. Secondly, when contact lenses contact with improper chemicals of eye drops, it may make the lenses dry out. Then you may get irritation in your eyes.
You should put the eye drops at least 5min before wearing contacts. Most eye drops contain preservatives and the pharmaceutical agent which may get stuck in the lens over time. So you should not use eye drops when wearing contacts.