I don't think you need to go to the ER unless anything swells, you get severe pain, you can't urinate, etc.
When you go to the doctor, tell them everything you mentioned here. Print your post out and hand it to them, maybe. I promise you that your doctor has heard about ED before, and while it is embarrassing for you, it's clinical for your doctor.
I can't say what's happening - low hormones, a nerve issue, something with this lump, or something else entirely. If anything starts swelling, you get sudden, severe pain, or a fever, or any redness, go to the ER, but otherwise you can wait until your appt.
Let us know what happens.