I’m of the mindset that you’re an adult and can choose to go to whatever church you want or no church at all. You don’t owe anyone an explanation. Will people talk? Maybe, but that’s their problem.
I do think it would be in your best interest in the long run to take ownership of your decisions. You call your relationship with the pastor’s son “a situationship” but let’s call a spade a spade: He was having an affair with you.
And don’t let him blame his “hypersexuality” on his molestation. He told you that as a deflection so you wouldn’t be mad at him. That’s really insulting to victims of sexual abuse.