Removing the condom when you are still erect is mostly to prevent semen from spilling into a vagina, to prevent pregnancy and infecting a female partner if you have any infections.
There would be little risk to you from fluids on the outside of the condom, as they would need to get inside your urethra, and that's unlikely.
You should make it a habit, though, to remove it while you're erect to prevent any accidents.
This has pretty good instructions - and not safe for work illustrations - https://www.cdc.gov/condomeffectiveness/external-condom-use.html