If you can have an orgasm with the toy, you're fone. It might be that direct clitoral stimulation is too intense for you. The toy might be putting more focus on the areas to the side of your clit, rather than directly on it.
There are more than 8,000 nerve endings in the clitoris, and every woman is different. Some prefer direct stimulation, others to the side, some a mix, etc.
For some women, emotional intimacy is very important for sexual pleasure. Some need that in order to relax and let themselves fully enjoy the moment. Some women don't need that. One isn't better than the other - it's just who we are. I don't know if that's what's happening to you - only you can determine that.
So play around with different touches on yourself - where you touch, the pressure and speed of it. Maybe you'll end up just preferring toys, which is okay, too. :)