Of course it’s not wrong to tell your girlfriend how you feel. In general, it’s usually not a bad thing to tell a significant other how you’re feeling; the problem is when you start placing demands on someone, using your feelings as justification for being controlling. But in your case, I don’t think it’s even a bad thing to say, “I’m really uncomfortable with your having an intimate photo shoot with someone you used to sleep with. I can’t tell you not to do it, but I can tell you I am not happy about it and it makes me question your regard for me and my feelings.”
Are there no other photographers who can do a boudoir photo shoot? Why does it have to be this guy? Finally, you say every time the subject comes up, your discussions turn into all-out yelling fights. If you are already expressing your concerns in what you claim is a “calm” manner and she’s truly going “berserk,” then you need to re-evaluate the relationship. If you aren’t so much expressing your concerns as much as you are just telling her not to do this shoot, then YOU need to work on your end of the communication issue you two have.
If you have a relationship/dating question I can help answer, you can send me your letters at firstname.lastname@example.org.