He’s a 55 year old man, not a teenager. Ask him out for a drink. Be bold or don’t, there are no ramifications. You’re already divorced. People will always talk in a small town. You can let it bother you or not.
You choose the adventure. I highly doubt he will be offended you are expressing interest. The worst that could happen is he’ll say no and be flattered! I would be flattered if a nice, attractive person 15 years younger than me expressed interest.
What do you want for him, ultimately? Why are you avoiding this question? Is it casual sex? A relationship? Stop worrying what anyone else will think unless it will really affect your life. Will it?
Seriously, what do you want from him? If you want to get to know him better, ask him out for a drink to get to know him better. It doesn’t have to be some huge sexual come on. You can have a drink, be one on one with him and maybe something Will Burt’s inside you both and maybe you’ll realize he has terrible breath and old school selfish misogynistic tendencies that lead him to be single. You never know.
Go forth into the wild beyond of asking him for a simple drink, or not. It’s not a huge deal if he says no. It won’t crush you socially.