If you are signed up for committee duties like idk teaching Sunday school or ushering or whatever, then you need to tell those people you can’t do those duties anymore.
Why do you need to meet with these pastors? Can’t you just let the people you have obligations to, if any, that you will be attending a different church going forward, and then just stop attending this church and try out new ones? This isn’t confession time, and the pastor doesn’t need to know his son is a womanizer.
If you do need to have this meeting, just give them as little info as possible, and keep repeating that you have appreciated being part of this community but as you’ve grown and changed you’ve realized you need to explore and find the next right place for yourself. End of story.