The hot water causes your blood vessels to dilate, thus dropping your blood pressure. This causes your heart rate to increase to compensate which can cause symptoms like you experience and many more. In your situation, it sounds like you would be walking into trouble getting into a hot tub. You are not crazy for listening to your body.