“When I was still married I someday asked him if he could help me with insurance matters during a birthday party, because he works for an insurance company. But then some people joined the conversation and it didn’t went further.”
Why would this have gone further? You were married and you asked about insurance.
Your friend actually isn’t a friend, so she’s a non-issue.
I’m sure you can find a time/place where you are in the presence of this man without your exes in the vicinity.
I say don’t do it. Every update you write makes me think this isn’t the move for you. You’re not open to being direct, even though it’ll just take one small but straightforward move to gauge his interest level. And if it has to be a secret? Just enjoy the crush.