Damn by the end of it I wanted to punch him! He really put you in a difficult spot by saying something so harsh. I could see where maybe he was joking, but from the sound of it doesn’t sound like an isolated incident. If all was like it was then maybe you could let it slide, but now things are different and that sort of served as a wakeup call. I don’t know if the passive-aggressiveness will work because he may out of laziness or fear of a fight pretend not to see it and then you two slowly end up more and more in separate parts of the house. I also don’t think you want to confront him head-on because you want him to choose to be more loving out of love and not guilt/fear. Maybe be up front about how “you feel”. Be vulnerable with him. I think if he’s a good guy and he sees you hurting and letting down your shield then he’ll want to step up to make you feel better.