Ladies, has your boyfriend compared you to their ex?

Out of nowhere, my boyfriend brought up a guy I used to like back in high school. He told me in a joking way, "no wonder he didn't go out with you, his ex was hotter." That really pissed me off...he kept saying that I was jealous of her and that made me even more angry. First of all, I don't care about the past... I just had a CRUSH on him... that was it! He made me feel insecure about the way I look so I said back to him..."well I f**** hotter guys than you" in a joking way of course. Well that pissed him off and what he said next was...."well you suck in bed.... my ex f***** way better than you." Those were his exact words so we started going at it...It just really hurt my feelings because he has a baby with his ex...was married to her... and yet he compares me to her. I don't know if I can ever get over what he said even though he said he didn't mean it.. He said he went too far... Now he keeps saying sorry, but it just crushed me emotionally and I don't know if I can stay with him anymore. We live together and I feel like this is something i can't forget. What should I do??