Do you talk to anyone about your anxiety? You should.
Yes, HPV can affect the eyes, but it is thought to be rare. The most common strains that affect the eyes are the low risk kind, the ones that give you visible symptoms.
When you get contacts, one of the things they teach you is to wash your hands prior to putting your contacts in, every time. If you make that part of your routine, you'll be fine.
Also, consider that HPV is the most common STI, by far. It's estimated that 90% will have it at least once in their lifetimes. With that number, and how many wear contacts, if it were a big concern, more people would have ocular HPV.
Please, consider talking to someone about your anxiety.