It kind of reminiscent of asking someone you're about to have sex with, if they have been exposed to an STD. What do you make of it if they say "No, I've been careful"? For this assurance to be of any assistance in protecting you, the person would have to be truthful, (and when sex is in the offing, not everyone is truthful), and the person would have to be correct, when maybe they have been exposed but don't know it. Covid is more invisible than STDs, but the issues are a lot the same, about trusting the person to know what they're saying and not to lie. Given what you have described, I'd be more concerned about an STD than Covid, but the point is, only have sex with people you know (not just hope) are totally responsible and good at self-protection.
If you were wearing a visor and a mask, you are probably fine, but do know that asking someone if they have covid is useless as we all by now should be well aware it can be passed by asymptomatic people who have no idea they have it and might never get any symptoms and so never know they had it. Can you have gotten it? Yes, because while you were pretty well protected, your partner was not and you were not socially distanced. Is it very likely? No.
If he had the virus, and if you were breathing hard, and if your face mask is not of the most protective kind, you could possibly have been exposed. Not from the touch as much as from breathing the virus in the air. But if he is not sick you were not at risk.