Daily contacts are cloudy by end of day?

[kded31]

Hello - I use daily contact lenses and I never wear them for more than one day but I noticed that by the afternoon or evening my left contact gets cloudy with buildup that is difficult to remove when I try to clean it. It is annoying and affects my vision and eye comfort. I do have astigmatism in my left eye and the buildup always occurs at the area where the astigmatism is (where the little line is on the contact). I have been wearing contacts for over a decade and this didn't use to happen before I used special lenses for astigmatism. This happens almost every day.
Is there a problem with my eye? Do I have the wrong lens for my curvature? Or is it something else?
Thank you!