Cant fight back due to confidence or weak barrier in my mind?

Why is this when someone annoys me i can never fight back or just hit them? I dont feel weak and im pretty confident with comebacks etc but i never can spit them out or use them on anyone easily..without needing to force myself to.