It could be herpes, or it could be a lot of other things.
Of course, the only way to know is to get tested. Getting a swab test now is pointless, since you've had it for so long, but you can get a type specific IgG blood test now, and if it's negative, get another one at 4 months.
I also don't think you need to test constantly for herpes, but I do think that if you have symptoms, you should test. I think that if you have an STI, you have a responsibility to help prevent the spread of it, and having mild symptoms makes no difference in the rates of spreading it. With herpes, you shed the virus at the same rate if you have milder symptoms or more severe ones.
You may have mild symptoms, but perhaps a partner you spread it to may not. You're having unprotected sex. Herpes isn't the only thing you're at risk for, so I'd advise you to get tested for gonorrhea, chlamydia, syphilis, HIV, mycoplasma, NGU and herpes.
If you do have herpes, it can increase your risk for getting other STIs, including HIV, so I think it's really important that you know.
I don't know if you have herpes or not - only testing will tell you that. I don't mean to sound harsh or judgemental at all - I just want to stress how important it is that you know. You're engaging in some pretty high risk activities, and I believe you should be totally informed about it. :)