So first, let me just say that the hsv1 IgG test misses a full 30% of infections, so even if your test was negative, you don't know for sure that you don't have it.
67% of the global population under 50 has it, and 90% of them will never have symptoms. If you have it already, the chances of you getting it again in another location are as close to zero as it gets.
You can get hsv1 - usually oral herpes, but can be transmitted to the genitals or anus - from rimming, but without a sore present the chances are a lot lower.
I wouldn't worry too much about this. I know that's easier said than done, but your chances are really low. If you go outside your marriage on a regular basis, insist that these things don't happen, and remember that guilt doesn't equal risk.