The culture is more accurate than the serology.
The blood test could be negative because it was too early to test, or because the hsv1 IgG misses 30% of hsv1 infections.
The good news is that with hsv1 genitally, it sheds infrequently, and recurs infrequently (many people never get another outbreak after the first), so you are unlikely to ever transmit it to anyone. Also, if someone has hsv1, even orally, they can't get it again from you.
Let me know if you have any questions.