This is kind of weird. Is this all happening over text only? Or are you spending time together IRL?
Like… it’s perfectly reasonable/normal for someone to not respond to a text for “a good few hours” when you’re busy without explaining yourself.
Anyway, nobody can force you to talk to someone you don’t want to. You aren’t stopping because for some reason you don’t want to, even though you acknowledge it’s not a good idea. If you want to stop talking to him, stop. You can simply not respond. You can block his number if you need to.
IMO you’re wasting your own time here.