That's a silly statement for a doctor to make. Once the healing begins, a herpes lesion looks like any other lesion and has no special characteristics.
Your skin could easily be from healing pimples, boils, blocked ducts, irritations from rubbing etc.
Thanks for the feedback. Do you know what else could have caused the skin to look that way? The doctor seemed to think that was the only probable thing it could be, even though I had no other symptoms.
No this doesn't sound like herpes sores.
The better way to understand your herpes status if you have concerns is a blood test for IgG antibodies for HSV1 and HSV2. I don't recommend this though.
If the swab is negative, let things lie or if you can't then mutually test with your partner.