At no time were you at risk of contracting HIV.
Your doctors opinion is stronger than any opinion on this forum. If you wnat a second medical opinion there is a doctors forum for a small fee. I would agree with your doctor that while the risk is very low, condoms are not 100% effective always. If you read any box of condoms it will tell you they do not eliminate risk entirely, mostly do to improper use. NOT A RISK is a term used too loosley in my opinion.