Maybe, you're too dry... try using a lubricant.
I guess you are not ready, and you should leave it to its time. I had this problem several time. For me I think it would be because of the size difference in one hand, and on the other hand, you want it but you are not ready physically.