I'm sorry you have this situation. Have you ever talked to a therapist about it? Phobias and such can sometimes be helped through exposure therapy. https://www.apa.org/ptsd-guideline/patients-and-families/exposure-therapy It helps break the patterns of avoidance and fear through slow exposure to the fear. I think it can be a little hard to think about that BUT a well trained psychologist in this type of therapy would know how to guide you and help you. Might be worth considering especially if it is causing panic attacks.
Fear of heights in all its forms is, I believe, the 2d most common phobia (the first is fear of speaking in public, officially, but I'm guessing more people confront heights than the need to speak in public so I'm guessing heights might be the most common fear). I have this phobia, but didn't have it until I got a full-blown anxiety disorder. If this is your only phobia, good; if you have several and are anxious about a lot of things, harder. One way to fix just one phobia is to force yourself to keep doing it over and over until the phobia goes away. This is harder if you have a lot of anxiety and a lot of phobias. If you just can't do that, then a good psychologist who specializes in the treatment of phobias should be able to help you out. Don't fret, pretty much everyone has at least one phobia. The only question is how much it interferes with your life, and since yours is one you will have a hard time avoiding in your life, therapy would probably be a good idea for you. But no, you're not alone on this, it's very common.