So this is the thing. Guys like this, they can’t be with a bigger girl, publicly. They are too insecure. They might actually like and be attracted to someone bigger, but because it makes them feel bad about themselves, they’ll make YOU feel bad about yourself. Ad nauseum, rinse and repeat. At best you’ll be a dirty secret. At worst he’ll just be openly cruel.
Don’t go back to being enemies with him, don’t be friends either, don’t confront him or make a big statement, just back away. Stop engaging. Fade. Guys like this are horrible for you. You shouldn’t have to work so hard to get respect and affection. Focus on people who are not jerks to you. People who are just nice.