Normally with things like that, you have to consume large amounts. Unless something changes (i.e. sudden abdominal pain, bleeding, etc.) then I don't think you'll have much to worry about. Keep going to the doctor regularly for your pregnancy and just avoid ginseng from now on, but I think you and the baby will be okay. Teas like that usually are not very strong. :)
Okay so I looked it up today because I was curious too.
Ginseng is perfectly safe.
Actually all ginseng is really used for is as a male tonic. It is good for men who have a low sperm count (plus it tastes pretty good as a tea).
The only times it should not be used is during acute illness and if you have high blood pressure.