I completely agree that what you describe is not a risk for HIV transmission. It takes penetrating intercourse of a vagina or anus without condom or sharing IV drug needles to transmit this fragile virus. And even then, it's actually pretty rare. Panties would be exposed to air. Air and saliva deactivate the virus. If you are in contact with feces, however, I would be careful of E coli and other bacteria. But as to HIV? No risk. Yes, you can move on.
I’ll also add that some dirty panties have come in contact with my exposed penis during the act, which also adds to the list of worries.
Your situation involves personal contact with an object in air ( panties, fluids, stool, maybe blood from wound etc.). No hiv worries, because you can't get hiv from personal contact except unprotected penetrating vaginal or anal, neither of which you did and you didn't share hollow needles to inject with which is the only other way to acquire hiv. Analysis of large numbers of infected people over the 40 years of hiv history has proven that people don't get hiv in the way you are worried is a risk.
HIV is a fragile virus in air or saliva and is effectively instantly dead in either air or saliva so the worst that could happen is dead virus rubbed you, and obviously anything which is dead cannot live again so you are good. Blood and cuts would not be relevant in your situation since the hiv has become effectively dead, so you don't have to worry about them to be sure that you are safe.
There is no reason for a person to test when they are safe. The advice took into consideration that the other person might be positive, so move on and enjoy life instead of thinking about this non-event. hiv prevention is straightforward since there are only 3 ways you can become infected, so next time you wonder if you had a risk, ask yourself this QUESTION. "Did I do any of the 3?" Then after you say "No, I didn't" you will know that it's time to move on back to your happy life.
I should correct it was an in home hiv mouth swab test for the gums not saliva