It depends. First, if you are boxing for fun, i think it would be OK. You can either take them off and swing away blindly or put in contacts and go. Second, if you are boxing for competition, don't even think of fighting with your opponent. You could suffer severe eye damage, then maybe it isn't such a good idea.
You had better not do that taking your eyes into consideration. We all know that a punch of thumb can hit you in the eye while boxing. If you do that, you will find that your contacts are easily to fall off, and then I think you will be embarrassed when you cannot see anything during the box. Of course, what I said does not mean that you can box if you are near sighted. As a friend, you can ask for help from your eye doctor who will help you to see clearly during the fighting and make you have a nice time. At the same time, you can make it by yourself, namely, you can have a try with some comfortable durable plastic frames or even prescription goggles with Croakies to keep them from falling off. Because of the two factors mentioned above, I do not think you will enjoy your fight while wearing your contacts lenses.
Well, your friend was right. It's really very unsafe when you are boxing. However, wearing contact lenses during boxing is much better than wearing glasses. That's because you can avoid being hurt by broken glasses which was hit into pieces by your opponents. By the way, keep your clean, don't let any sweat enter your eye when you are wearing contact lenses.