I'm sorry you're going through this.
Did your doctor do any testing, or just a visual diagnosis?
Have you ever had a cold sore on your mouth? It's possible - but not really likely - that if you have oral herpes, which is what cold sores are - that you could have transmitted it from your mouth to your penis if your hands touched your mouth, then your penis. However, if you've had oral herpes for a long time - longer than a few months - then you have antibodies that prevent you from getting it in a new location.
If your doctor only did a visual diagnosis, it could be wrong. Visual diagnoses are wrong a lot.