How late are you for your period?
When you took the test 3 weeks after sex, where were you in your cycle? Do you know when the first day of your last period was?
A pregnancy test is most accurate when you wait at least a week after you're late for your period. https://www.healthline.com/health/pregnancy/five-signs-to-take-pregnancy-test#when-to-test
It sounds as though your periods are irregular, though? If that's the case, and since you used a condom, you probably aren't pregnant.
I really think you are worrying for nothing personally. He used a condom and it didn't break. When they say there are condom failures that lead to pregnancy, it usually means that the condom wasn't used correctly or broke. I would say that by 3 weeks after sex, you'd get a positive pregnancy test or at least most women would. I used the early detection pregnancy tests and got a positive 4 days before my period was due. All it takes is picking up that hormone to trigger a test. If you track your period, you can test the day you miss it and then I'd wait a full week after to test again. But in all honesty, I would be shocked if you were pregnant after using a condom that didn't break. Keep using those!