If you get infected with HPV (which is highly transmissible between men that have anal sex), of course you can then infect your partner by having vaginal sex. You used condom when having sex with the other man, which reduces the risk, but it is not zero, as the contact skin to skin with the lesions caused by HPV can cause transmission as well. If you think you will repeat this kind of exposure, please consider the idea of getting vaccinated against HPV. It is highly recommended for men that have sex with other men. In so many countries, specially in Europe, the vaccine is offered for free by the public sanitary system.
I hope this helps.