Oh hon. I'm sorry you worry so much about this.
First, hands never transmit any STDs, including herpes. He could finger you and then masturbate himself and not give himself herpes.
Also, it takes more than just a touch of his penis to your labia to transmit herpes. It takes some friction, usually the kind with penetration.
I don't know if you are aware of the stats of transmission of hsv2. You may not be, since you've had it so long.
In 2004, a big study was done, and we got transmission stats.
Ghsv2 transmission, female to male, over the course of a year, assuming sex 2-3 times a week:
Only avoiding sex during an outbreak - 4-5%
Adding condoms OR daily suppression - 2-3%
Adding condoms AND daily suppression - 1-2%
You can read more about this in the Herpes Handbook, written by Terri Warren, who is one of the world's leading experts. You can find it here - https://westoverheights.com/herpes/the-updated-herpes-handbook/ It's free, too. :)
If you take nothing else away from this, please know you aren't some kind of walking contagion or biohazard, and your partner wasn't at risk.