for a 30 something man, 320 is very low. Most insurances would call anything below 400 as hypogondal for ANY age. I would expect a total T level for a 30 something to be well over 600.
Remember that a person loses about 1% of total T for every year between the age of 40 to 60. So it is doubtful that things will improve much by doing nothing.
18% body fat is pretty good shape. The more overweight a person is, the more fat they have, the more of Testosterone is converted to estrogen. I doubt you have this pr oblem at 18% body fat.
you need a full screening, to include LH and FSH as well as SHBG and total and free Testosterone. That will help you know what next options there are.
Get tested for thyroid to include TSH, Free T4 and Free T3 just to rule out thyroid as being a contributing factor.
Second opinion is really a good thing, I think you should visit another specialist. So far, according to test results, we can say that the level of testosterone is not lower than the low limit of the norm, but each person has a different rate. Poor muscle gain can be caused by insufficient nutrients in the body, mainly protein. It is not enough to train, it is also necessary to eat well. In any case, I would advise you to consult an endocrinologist, because low libido and increased fatigue can be associated not only with low testosterone production, but also with disorders in the thyroid gland. Take tests for thyroid hormones and perform an ultrasound exam.
Hard to say what is going on. To be honest, your testosterone level is inside of normal. Normal levels are 280 to 1,100 ng/dl's. You are sort of on the low end but not below. Let me ask you about this -- could you have a low level of depression?
I always think a second opinion makes sense when you have a worry. So, in this case, I'd consider it. and yes, mention your symptoms.
Here is some information on increasing testosterone naturally. https://www.webmd.com/men/features/can-you-boost-testosterone-naturally#1 I've also heard eating protein, carb and essential fatty acids can help increase levels.
I know that isn't much help. But it's a start.