Did your doctor initially diagnose you with hsv2 visually, or did he do tests?
If it was visual only, believe the blood tests. Visual diagnoses are wrong quite often.
If he swabbed you, and you have a swab that says herpes was detected, then I'd wonder if you have genital hsv1, which is missed 30% of the time on the IgG.