Do you have anal herpes? Have you been officially diagnosed? If you were, how were you diagnosed?
The thing about herpes is that it infects nerve groups. Genital herpes infects the sacral ganglia, which affects the area that boxer shorts cover, essentially. This means that you can get outbreaks in that area, and are infectious from that area.
You shed the virus from your penis and anus, so even if you only get outbreaks anally, you could still transmit the virus by having penetrative sex where you are the penetrative partner.