Are you sure they are fibers, as in, from fabric? If so, have you checked the insides of your bra with a magnifying glass, to make sure the fabric or the liner is not eroding?
When I was nursing, I would get lots of little dried milk bits in the wrinkles and crevices of the tip of the nipple. It's not even obvious that nipples have crevices unless you look with a magnifying mirror, but if I would break off a Q-tip (so the soft white cardboard stick was the end, not the cotton) and (watching in a magnifying mirror) slightly dig into the nipple, up would come a lot of those, and the end of the nipple would feel better after I did that. It kind of acted like paste, probably kept the nipple and its ducts from being able to naturally sweat and shed off skin cells, etc.
Possibly you're getting something similar, not with breast milk but with skin oils and salts. Even when I wasn't nursing, using that technique sometimes if my nipple felt a little painful or blocked, there was always some salt or dried oil or whatever it is, in the crevices. If you are large busted, it might be even more likely that sweat and skin oil could play a role. If you decide to try to clean out the crevices, use something gentle, like the (white cardboard) stick of a Q-tip, or maybe the end of one of those toothpicks that has a square end (not the pointy end), and just gently run it across the nipple and see if something comes loose. If what is happening is that the nipple is getting blocked by natural oils, dead skin cells and salts, no wonder it might get irritated over time.
Sounds like Morgellons Disease to me. And that is one VERY mysterious disease, with a lot of controversy about what it actually is.