Panic and anxiety can occur even when nothing stressful is going on. In some people (and their families--there does seem to be a genetic factor), it appears to be a built-in feature, sort of like a tendency to have certain allergies.
There is no shame or blame or even a problem with self-control. It is simply a medical condition, and in my experience, it can be treated very effectively.
If you have a general practitioner, he or she can refer you to a psychiatrist or a counselor. I would recommend this over being treated by a GP, because panic really is a specialist's job, and some medication may be necessary. However, starting with a GP would be better than nothing.
Really, with a few sessions of cognitive-behavioral therapy and maybe six weeks on a low dose of an SSRI like Zoloft, you will get your life back and feel a lot better.