By all means YES...cheese, milk and all commercial dairy products should be avoided.You are just clogging your system and introducing toxins like hormones and antibiotics to your already weak system.
Google 'dairy products and cancer' and you will see how many websites are against eating dairy products when one has cancer cancer. But there are some websides that will call it quackery. Read them too and compare.
Dairy products are bad for you anyways...the human body doesn't want to digest something made for a baby cow with four stomachs.
If you do want to keep drinking milk, always go for the organic brands that don't have any added hormones, etc. to them. Silk Soy Milk is an excellent substitute and tastes great.
Beyond that, there is no concrete reason to avoid it. I think you may be concerned about a 2004/2005 study about this. Here is a link on WebMD about that:
http://www.****.***