And, just so you have the relations correct, YOU are related, he is your uncle. She is not related by blood only marriage if she is still married.
A few generations ago it was not uncommon for a brother of a fallen to marry his widow or the sister of a fallen sister to marry her husband.
Incest means sex between blood relatives. If your mom is having an affair with her husband's (or ex-husband's) brother, it's not incest, but it's kind of tough on the brotherly relationship between your dad and his brother. Are you saying she is still married to your dad? If not, and if she is unmarried, she is within her rights (as long as you are not a small child who is being ignored while she goes out and has fun) to date whomever she wants, except her own blood relatives. Whether it is a good idea is another matter.