Sure, that approach might work. But I wouldn’t say “it was a joke” – because it will block any seed you’ve planted. Better to shrug with the implied “your loss, fool” than tell him you’re fucking with him.
I really think you’re over thinking this and trying to twist yourself into knots to make this risk-free. Make some noise! Take a risk!