Am i crazy? Am I being too desperate not to walk away form him?

I have never been an insecure person but the way he asks is taking a toll on my self esteem! Specs: Dating a year, very involved with his family, daughter, makes plans for vaca next month etc. Problem: Had a pool party to go to for his daughter's tball team. Didn't invite me. was very vague when i text him, drank moonshine, wanted to stay longer but daughters mother wouldn't let her, we had a church stew that night, all I hear is how freakn tired he is the whole time. That's all I hear. either he is sick or he is tired but never in a good mood when I am around. very rarely is he affectionate and never compliments or says nice things, its always negative about me..I let people walk all over me, i suck as a mother because I don't beat my child, etc. why do I still stay?? I don't trust him, but when he is sweet he is so wonderful! I have never thought I would be caught up in this kind of relationship?