Does this sound like what you have?
https://www.austinvasectomycenter.com/treatment-for-adults-with-retractile-testicles-causing-discomfort/ (I'm not endorsing this doctor, the center or the procedure - it's one of the few articles I could find on it in adults.)
It seems to be a condition that affects mostly younger boys, but adult men can get it, too.
Here's an article describing it in younger boys -
In any case, you definitely need to see a urologist. It sounds like a simple procedure to fix it, but get it evaluated by a urologist to make sure that's what's happening. Let us know what happens. :)