I think your very closest family - like your parents or siblings, would be supportive if they knew you were trying but were having fertility issues. You don't have to get into the details, but just say we're trying and so far haven't been successful.
Everyone else, just smile and say when we decide to get pregnant again, we'll let you know.
Amazing, how nosy people can be.
People need to mind their own business. I agree, tell your immediate family such a parents and siblings as they can be support for you in these tough times but as for everyone else- it's NONE of their business. I'm on a rampage this week because my little guy who has delays and isn't talking even though he has speech therapy 5 times a week and we are constantly making him try to say things (he was diagnosed with Apraxia) was at a restaurant with us this week and a lady whom was just waiting to get a table like we were says to me "Is he not talking yet because you always talk for him?" I was FURIOUS. If she only knew how much we work with him and how hard it is for him (he's 3 1/2). People just need to mind their own business. I'm sure this is a very difficult time for you and the last thing you need is people bugging you. I've learned to say "I'm sure you didn't mean that to come out that way." (and boy do they back off) but I don't know if that would work in your situation. Maybe just say,like Rock Rose said above, "When we're ready" and you can always add that it's none of their business. I bet they don't ask again if you add that. Good luck as I know it breaks your hear every time it's brought up and you don't need that. Take care!
I agree with the answers above, but I'm curious- after 4 years of TTC, have you considered a referral to a specialist?