He is hurting me even mo, what should i do?

I broke up with my boyfriend about 3 weeks ago. Sometimes he calls me saying he wants us to work things out and i know i love him too. But then we broke up because of his ex who is a sugar mamma keeps getting involved in our relationship. I tried to avoid him by not answering his calls or reply his messages but he comes and knock on my door. I refused to sleep with him ever since we broke up cos it wont make sense to me. Everytime we talk we end up fighting to an extend were he told me last time he doesnt care about me and he doesnt have an feeling for me watsoever, becos i had told him to back off since he is confusing me. Now i went to a party his friends are friends with my best friend, we went to a party together he heard about it. He got so mad he phoned me shouting at me. I'm getting stressed out with tho whole thing. I still love him, but still i cant just run back to him when things are not clear like this. what should i do?