You have to have a cut or abrasion for herpes to get in. The reason why it is often sexually transmitted is simply because mucous membranes are more delicate than ordinary skin and so get damaged, cut or abraded more easily. So, no you can't get herpes just from touching it with a finger that has no open wound. however, you should wash hands immediately with soap and water to avoid spreading it to another part of your body or someone else's.
First, his sore could have been a pimple or ingrown hair - not everything that happens in the genital area is herpes, even in someone who has it.
In any case, if it was herpes, a simple touch isn't going to transmit it. The skin on your hands is too thick for the virus to penetrate, and it takes some friction for it to even transmit to mucous membranes (mouth, genitals).
I'm sure that if you had that kind of friction, it would have hurt him, and he'd have stopped you.
You don't have anything to worry about here. :)