If you have had hsv1 for longer than a few months, then you have antibodies that will help protect you from getting it anywhere else on your body, especially in the absence of an outbreak, when there is less virus on your skin, or in any saliva that may pass over your skin.
You should obviously be careful about touching an outbreak and touching your eyes, but what you are describing shouldn't be a problem.