Its fine and you can clean the litter box as well. they say you shouldnt clean the litter box because of toxoplasmosis, the bacteria found in cats stool. They can have that bacteria if they eat raw meat or mice or whatever. If they are inside cats they cant have that bacteria. regardless of if they are inside or outside cats, just wash your hands properly and you'll be fine. The only way you can get it is if you eat the poop.
Just stay clear of the litter box. We have a cat.
Yes just don't clean the litter