No. As long as you get used to toric contact lenses, it won't make you feel uncomfortable. Toric contact lenses need more expertise to fit one's eyes exactly than regular contacts. So the eye doctor should be experienced enough to make toric contact lenses. Otherwise the toric contact lenses won't fit you properly. If the toric contact lenses don't stay in place, you will feel uncomfortable because you will get blurry vision when moving eyes or blinking. But Once the toric contact lenses fit you exactly, you will feel comfortable just like regular ones.
I never wear torics contact lenses. But my one of my friend has worn such contacts. He bought torics contact lenses according to your prescription, and the torics contact lenses give him comfortable wearing. Now, he have been wearing the contact lenses for years. But you always keep your eyes are moist because eye drys may cause the contacts slight off the proper places that cause you vision blur.