Okay so I looked it up today because I was curious too.
Ginseng is perfectly safe.
Actually all ginseng is really used for is as a male tonic. It is good for men who have a low sperm count (plus it tastes pretty good as a tea).
The only times it should not be used is during acute illness and if you have high blood pressure.
Normally with things like that, you have to consume large amounts. Unless something changes (i.e. sudden abdominal pain, bleeding, etc.) then I don't think you'll have much to worry about. Keep going to the doctor regularly for your pregnancy and just avoid ginseng from now on, but I think you and the baby will be okay. Teas like that usually are not very strong. :)