It might be a cleaner issue (and possibly easier for you to forgive and try again) if he had just been talking to her, but for him to then turn around and accuse you of cheating makes his behavior a lot more hypocritical. In your shoes I'd probably be madder at the false accusation than at him having a few words with his ex. But gotta say, none of this sounds like it's heading in a good place long term.
Hey, I'm really sorry this happened! That hurts to the core to be betrayed like that. Sadly, I don't think you can deny that this is a situation where he's been unfaithful. Even if it was just for ya ya's like phone sex, it's still way too intimate for a guy in a relationship! And his reaction is such that he got caught and was grasping as straws about what to do, I imagine. So, he went with the turn it all around routine. People do this sometimes and just make things worse!
Unfortunately, unless he is willing to really talk to you and admit what the truth is, there isn't much you can do. He's not saying sorry, let's work on it babe. That's what you want to hear if you hope to fix this situation and continue a relationship with him.
How is the rest of your relationship? Does it seem somewhat one sided or has it otherwise been healthy? Did he cheat on his ex with you or someone else -- as in does he have a pattern of this kind of thing?
I don't know how much you have invested in him. If your really committed you may want to check out BAN. Beyond Affairs Network. They do a group thing that might be of value. Talking with other people who have been cheated on can help you talk out your problem and possibly help you make a decision on where you want to go. If you do decide to try to work it out one other thing I would suggest you consider is therapy of some kind. Couples or individual. Maybe a couple sessions of individual if your agonizing over the direction you take it.
Hope it helped!
The last thing I might mention is that you've been using unusual language when you have been saying you love him. You haven't said "I love him." You've added a little qualifier each time. "I love him dearly" doesn't quite mean the same thing, and "I love him and all" doesn't mean the same thing. Maybe it just reflects your doubt based on his behavior, or maybe it reflects you having second thoughts in general about how good this relationship is, but whatever is going on, if you intend to stay with it, it would be good to see a counselor and figure out what you want and need and what it will take from him for you to stay. You sound like you can stand on your own two feet and take care of yourself fine, and like you deserve someone who is in it for the long haul, not some guy with an eye off to the side. There are lots of men in the world who wouldn't do that to the woman they are with.
Well, everyone I love your guys advice, I have let it go and tried to move forward with him and put that and our other problems in the past and leave them tgrrey, but I guess it wasn't meant to be. He doesn't want to move forward with me. I love him I always will, but I guess im just not the one. Again thank you all!!