Should i stop my ex seeing our daughter?

i split up from my husband in 2005. we have a 6year old little girl. my ex has been terrible abusive and just nasty to a point where the police were pretty much on speed dial. Then as time went by my ex grew up and moved on, and away.... he doesnt pay any maintenance and contact has been difficult. one month to the next. since this year he has seen our daughter for 6hrs. he hasnt called text spoke or tried to see her. he ignored her birthday. then every so often he reappears demanding to see her. so i agree and he lets her down again and again. My daughter is a level headed little thing but for a couple of weeks after she sees him. she is totally out of sorts, she wont go to bed she crys at anything. hates to be left alone. she even says now that she doesnt want to see daddy. I just hate the fact that he demands to see her when he wants to. he doesnt take in to account what my daughter wants or feels. because this and the way it affects her i want to stop him from seeing her. can i?