He is just paranoid that you are trying to get even with him. The only thing you can do is reassure him that you didn't cheat on him and that the baby is, no question, his. If he doesn't get it, there are only so many ways you can tell a person something. Either he will accept it or he won't. You shouldn't have to shoot for the stars to get him to believe you.
He is probably saying that because he is feeling guilty because of what he did, and now he wants to make you feel bad also.
Can you sort things out between the two of you, maybe try counseling. If not, and if you fight a lot, think of your children, they shouldn't have to grow up in a house with constant fighting.
You have been blessed with another little miracle inside you and you should take care of yourself, baby, and your other children. Do what is right for you.
Stress is not good for you and the baby.
Good luck with everything!