I am sure you are going to be fine- I can understand your concern, but I've been exposed to homeless people my whole life- my father was homeless for some time when I was an adult, my husband and kids have gone to serve meals to homeless frequently- we've never caught anything. You aren't usually going to catch something by just touching them, except maybe a cold- which you can catch from the richest man on the planet by shaking his hand too. After you have contact with any person, it's ok to wash your hands- especially during cold and flu season, but there is no reason for you to worry more about your encounter with this man, than any other. Hope that helps!
Just wanted to check to see how you were doing? Hope that your mind is still at ease about this!