Your situation involves personal contact with an object in air (fluids, vagina, clothes, body, your fingers etc. which is not a risk for hiv.). No hiv worries, because you can't get hiv from personal contact except unprotected penetrating vaginal or anal, neither of which you did and you didn't share hollow needles to inject with which is the only other way to acquire hiv. Analysis of large numbers of infected people over the 40 years of hiv history has proven that people don't get hiv in the way you are worried is a risk.
HIV is a fragile virus in air or saliva and is effectively instantly dead in either air or saliva so the worst that could happen is dead virus rubbed you, and obviously anything which is dead cannot live again so you are good. Blood and cuts would not be relevant in your situation since the hiv has become effectively dead, so you don't have to worry about them to be sure that you are safe.
There is no reason for a person to test when they are safe. The advice took into consideration that the other person might be positive, so move on and enjoy life instead of thinking about this non-event. Next time you wonder if you had a risk, ask yourself if you did any of the 3, then after you say "Nope, I didn't" then it's time to move on back to your happy life.
1.Unprotected vaginal or anal sex
2.Share of IV needles
If you did not have any of the above two exposures, 'thou shalt' not worry about HIV. What you describe above is casual contact which is no risk to infection or transmission.