There is no risk here. First, herpes requires direct skin to skin contact in order to transmit - you got ghsv1 from oral sex - someone's mouth was directly on your genitals.
Hsv2 is transmitted by direct unclothed genital to genital or genital to anal contact - usually penetrative sex.
Like you said, there's movement - wiping his hand on the bed, exposure to air, time, etc.
Also, if your partner already has hsv1 - and a large portion of the population does - he can't get it again genitally.
I'm not sure you know this, but you are unlikely to ever transmit this to anyone. Ghsv1 rarely sheds, rarely recurs (meaning you don't get a lot of outbreaks), so unless you have an outbreak, you probably won't transmit it. After about 2 years, you'll be shedding about 4 days a year.
There's debate on whether or not you should tell your partners since the transmission risk is so low. I vote for telling, because it's a trust thing. If a partner ends up a relationship, and you hadn't shared the info, it wouldn't be great if they found out months into the relationship. It also opens the discussion about STDs. You need to know what they might have, too.
As far as this encounter though, it's fine. :)