Listen to your doctor, that is the truth and what you're experiencing is another normal part of aging. Male sexual peak is between 15 and 30 then gradually declines from then on.
A reduction in semen can be caused by certain drugs (such as Flomax and others). It is somewhat a part of aging but can also be caused by prostate enlargement which interferes with proper ejaculation just as it does urination. A small amount of a daily alpha-blocker ( a blood pressure med) can help relax the prostate muscles, if that is causing a restriction. If your doctor dismisses it, insist that this is bothersome to you and make sure you are doing all you can.