I think your suspicion is correct - I'm pretty sure this is in fact semen or prostatic fluid. First, it's not all that uncommon for some men to expel a small amount of prostatic fluid or semen during a bowel movement. Remember, the prostate can be reached (either for an exam or for stimulation) thru the rectum. And since you haven't been sexually active recently, there's almost certainly a buildup of prostate secretions & semen, and the extra pressure in the anal area is probably enough to sort of 'squeeze' the prostate & cause this fluid to escape.
This is not harmful in any way, altho I would suggest if you're not having sex, you should be stimulating yourself to orgasm at least 2-3 times a week. It's not healthy for the prostate to allow it to become 'congested' by a buildup of fluids which eventually become somewhat 'stale'. This can lead to prostate issues down the road - either prostatitis in the near term, or even worse, many years down the road, prostate cancer. There are a number of studies showing that guys who ejaculate more frequently (from either sex or masturbation) have a lower risk of prostate cancer years later. The theory is, allowing these 'stale' secretions to accumulate & stay in the prostate produces bad substances that irritate the gland, which could lead to cancer later on. It's also healthier for the prostate to have a somewhat regular 'schedule' of ejaculation - a few times a week, but kind of spread out. So don't go a full week without & then go numerous times over the course of a day or 2. Instead, have 3, 4 or 5 spread out over the week. That doesn't mean you can't occasionally have 2 or 3 in one day, just don't overdo it. Try this & see if it eliminates this 'problem', which isn't really as much of a problem as it is just a symptom of your situation.
As an aside, younger guys usually have nocturnal emissions (so-called 'wet dreams') that expel the extra built-up semen during sleep, but frequent ejaculation lessens that effect, and as guys get older, the wet dreams usually diminish, so you may need to 'clear the pipes' fairly frequently yourself, which is not so bad as medical advice goes, right ??!!