This man isn’t interested. You gave HIM a sign by following him outside. He could then have started talking to you, in a way that would have looked friendly only if anyone came outside. He chose not to. This might indicate that, like your ex said, he knows you like him and doesn’t want to encourage it.
Even if there were mutual attraction, it’s clear that you two could never have a relationship. Time to move on and meet men outside the family circle. Snap out of it.