This is a tough decision that ultimately only you can make. The fact is that most mosquitoes aren't carrying Zika, so it's not a huge risk. Also, they primarily come out in the evening, after sundown. You can wear long sleeves if you're out then, and long pants, and use repellent on the exposed skin. I'm sure that's what the doctors meant by "be careful."
But, if you're going to freak out if you get a mosquito bite, then it's probably best not to go. If your vacation will be ruined by that anxiety, it's not really worth it.