You're worried because NOTHING has gone wrong... Yes, you're crazy.
MOST people have children that are born healthy. You are trying to create a problem where there isn't one.
There is no reason for you to be scared. And sleeping with a child won't stop anything from happening. My daughter died of SIDS, while wide awake and in her doctor's arms. It's like being struck by lightning - there's nothing you can do, and no way to prevent it, so as long as you exercise reasonable care, there's no point in worrying about the inevitable. It's like worrying about the end of the world - it's completely out of your hands, so just go ahead and live your life. Worrying won't stop anything.
My aunt had 12 kids - all perfectly healthy and now grown and with families of their own. You can't jump to conclusions or try to predict the future.
Go ahead and have another. You have a great track record, might as well capitalize on it. I'd wish you good luck, but it seems you already have plenty of it!