Semen is not good for a phone, nor is soap and water afterwards during cleanup, but a damp cloth might be okay. My advice is to stop using the phone for masturbation.
You say "my mom and sister use this so now I am worried." If you are worried that they will notice guck on the telephone and you will have to explain it, hey, just tell them you spilled something on it and were trying to clean it. You might have to listen to a lecture about not getting the phone wet, but you don't have to tell them what you were doing. If you are now worried because you are spinning a fantasy in which your mighty sperm is dangerous to the women in the household, please remember the facts from your Human Sexuality class and don't let yourself grow up to be like this crazy guy who every now and then posts in with elaborate fantasies about his mighty semen impregnating his female relatives -- know that sperm does not survive exposure to oxygen, and just because there are females in the house, you masturbating does not put them at risk of pregnancy. Seriously, this does come up in posts on this site, from people who are somewhat conflicted about masturbating or want to feel more powerful in relation to their female relatives, and it's just stupid. Don't let yourself go there, it's a road to goofiness. Take care.