Agree with CML2014...................tell him what you've posted here.
Good luck.
.......and what is wrong with you telling him exactly what you posted here.
His behavior is threatening the good friendship you had. He needs to know you have a life that includes sharing time with OTHER people not just him.
I can tell you this if you allow this to go on without talking about your concerns GOOD FRIENDS will not be the term you will use to describe this relationship in a very short time.
Talk to him NOW! You have every right to your feelings on this, he makes fun of you to intimidate you--yea, that's a real nice friend you got there.
CML