For that doctor to blab autism off with no testing or other information besides your self report was off target! He should to his homework first. Autistic children take normal input and reinterpret into some personal meaning most of us cannot understand. What was your history in school like? Can you carry on a conversation and respond appropriately? Are you able to take care of yourself? Do you have any friends or family who could substatiate anything the doctor said? What is your family history?
I was trying to figure out what my 11-year old son was thinking. He thinks in 3D, able to dissect vehicles and buildings from top, bottom, and side views. I read a book, THE GIFT OF DYSLEXIA by Ronald Davis. It explains much of this way of thinking, including seeing colors. Dyslexia is not just having trouble reading, rather it is a whole different way of percieving things. Some artists report that they see colors that they cannot replicate in their paintings. Who cares if you see colors mixed in with numbers, as long as the math makes sense! When I told my husband what I was reading in Ron Davis' book, he said, "What? You can't see in 3D?" It never occurred to him that others couldn't see what he saw. The doctor cannot imagine anyone thinking 3D because he doesn't see that way.
Nothing is wrong with you. You have a gift that you can use in many pursuits where the rest of us (flat thinkers) fail.
A diagnosis of autism is given based on significant deficits in language and social development and the presence of restrictive/repetitive patterns of behavior. Criteria for Asperger's Disorder also include social difficulties. Your description does not seem to fit either diagnostic category. The best approach would be to set up an appointment with a psychiatrist who could conduct or arrange for an evaluation.
Your doctor should be able to refer you to a multi disciplinary team that specialises in autistic spectrum disorders such as autism/aspergers etc. They should be able to assess you and give you some insight into your condition. You really need help to find out what your strengths and weaknesses are. Do you have difficulties with social interaction for example?
As you mention colour linking into number associations etc it might be worth you googling synesthia to see if that sounds like what is happening to you. This is when sensory information can get mixed up and associated with other senses. For example you talk about numbers being related to colours. Other people might have colours that relate to smells. It might be that you have synesthesia and some autistic traits, but not enough for a diagnosis.
I am not a doctor, I am a parent of a child on the autistic spectrum. But I think you are right to be looking for answers to your experiences.