first of all, nitrous canisters are legal to use for making whipped cream, to be administered by a dentist, or for automotive purposes, not for cracking and getting high. i know plenty of people who have been arrested at "tank parties" or at concerts for abusing nitrous.
second, nitrous kills your brain cells pretty efficiently. after a year or two of nitrous abuse you can see some serious mental degradation.
that being said, you won't get significant loss of brain function from one nitrous cracker. your doctor is probably right, so don't worry, but don't mess with that anymore. it is seriously the drug of choice for idiots.