It depends what you want from your relationship. If your still in your teens, it's early days and there are less problematic boys out there. If you're an adult, go out as a foursome and see how that goes. Get some feedback from the other couple.
You shouldn't be struggling with this alone if he is a narcissist, but this may be a word you've coeme across and am labelling him this way when he's just said a couple of boastful things. Beware of labels! However, if he does have this problem then you have to decide whether you can love him as he is. You won't change him. So accept him as he is, or change partners.
If you are dating a narcissist and his behavior crosses the line, then yes, you should break-up with him. If he is making you feel bad about yourself all the time and disrespecting you, should end it immediately. The longer that you stay with a narcissist, the more likely it is that you will make excuses for his behavior and fall victim to his manipulation. If you do, you are falling into his trap.
Also, the more pain that a narcissist can inflict upon his partner, the less they will respect you. When you decide to get out of a relationship, don't let the narcissist charm their way back into your heart, which is a deceptive tactic that the narcissist uses to keep you in an unhealthy situation.