He needs to handle this. His parents, his problem. If he can’t stand up for what you need without attributing blame on you in front of his parents, that’s a big issue. A husband issue, not a parents issue.
I’d ask him to make clear boundaries and/or ask for the keys back. If this was your parents, I’d tell you to sit down with your husband, figure out what you want and need as a couple and you handle it. Don’t go as a couple, he should just do it. If you can’t trust him to do this… counseling?