Your doctor is a moron.
Yes, there is a chance for false positives, but that's manageable, because we know this, and we know that anything under a certain number on the test can be a false positive, and you can get that confirmed with another type of test. OR he can just start with the other type of test, like a BioKit or Western Blot instead of the IgG.
Your symptoms don't sound like herpes (and please, stop popping and digging at them - it's an easy way to cause infection). Loads of things can cause bumps in the genital area.
It's hard to say if you could have had only 2 outbreaks in all this time - some people get infrequent outbreaks, others get a lot. That's not really a determining factor. Cultures at 2.5 days could be accurate, but you'd been digging and popping, so probably not.
Just ask for the IgG blood test anyway. Tell him that you're a grown up and can handle the possible false positive, knowing it's a possibility. If the result is under a 3.5, it needs to be confirmed. (And this can happen to anyone, not just those with hsv1.) It may come back as clearly negative, and your doctor will have allowed you to waste all this time worrying because he's an idiot.