At this point you’re in control here. You’ve essentially been pursuing him. You made the first call. You invited him to your place after meeting up. You made the second call. You reach out when you’re visiting. Do you reach out between visits?
If you want something more, or you want something less, then you have to spell it out. This guy does not have ESP.