Your swab result should be taken as definitive. Swabs look for viral dna of the infection and are accurate.
As for your blood test, when you had it done with your std testing are you sure HSV was included? In a standard STD panel, hsv isn’t usually included. Even when people will ask for everything, unless it’s specifically requested, some doctors don’t include HSV.
Also, did your boyfriend have an hsv test? You may want to confirm that both of you had previous hsv tests.
Even without symptoms, hsv can be passed. Many people don’t have any symptoms at all.
Confirm your testing and that of your boyfriend to see if hsv was included and if so, what test was done. IgG? IGM?
I am not an expert but would like to provide feedback from my own experience. I was visually diagnosed with herpes in 2008. No test was done. My symptoms were minor. Every year thereafter or sometimes multiple times per year, I would get the full STD panel from STD check with the latest tests for HSV. I have never had a test come back positive. Every test came back negative or non-reactive. Last June had some minor symptoms and had a swab done. Positive. So One blood test after still negative. So in my experience, the swab is definitive and the blood test can be inaccurate.