If you do fingering, the hands must be scrupulously clean, including under the fingernails. Hands carry a lot of bacteria.
As long as you are gentle with yourself and your hands are clean as anniebrooke says (just wash your hands before hand or use hand sanitizer), it's not a problem.