It's nice when you can have continuity, but you may want to consider moving on to another doctor if your present one's bedside manner leaves something to be desired.
I have rest days, but at my age I have seen people younger than me work out less and have more results than me. My arms are still slim and my legs still skinny despite squatting well and many other workouts. I strongly beliieve there is something physically holding me back. I've also heard having high prolactin can effect testicle function, .is it a coincidence my package is extremly small, i have varicose veins, a cysts and high prolactin? I think all of those are a sign of something not working properly. I hate my endonxronologists, only met with him once and he was extremly rude. I have another appointment with him in a couple months.
As you've related, visiting a doctor can be both helpful and/or disheartening. As far as your main concern, penis size, you are well within the bounds of normality in erection. Any other health issues should be dealt with. You say you work out a lot and have always worked out. This may be off track, but I wonder if you are overdoing it or at least not allowing your body to recuperate between sessions, thereby affecting your growth..