You don’t mention your religion — and TBH I doubt I’d know much about your church even if you did — but yes, a reminder that even if it is the kind of environment where people customarily only leave a church when they “marry out,” you don’t owe anyone details about why you’re leaving. Even if this church and your religion are/have been a big part of your life.
Everyone else has great advice, I’m mostly just chiming in to say that it sounds like you haven’t fully dealt with the emotional baggage from your five-year affair with the pastor’s son, even though it sounds like it has been at least a few years since that ended. Your second post also mentioned that you’re seeking healing. Therapy! Therapy can be help you process that experience and why you let that situation continue for five(!) years knowing he was lying to both of you.