I would definitely worry about this and your ongoing use in spite of the problem. Now you've got an open area inside your nose which is a perfect breeding ground for infection. You don't want an infection that close to your brain.
You need to stop putting stuff up your nose. Period. You wouldn't believe how much pain this can cause if untreated. Put some neosporin type ointment up there on both sides of the septum, with clean fingers, and then keep your hands away from your nose.