How would you feel about this man being around your child?

Today I was at a street party, and the children were running up and down the street (a friend lives on the street). My daughter (7) ran up to me and said "mummy, we're (her friend who lives on the street and my daughter) are going to a mans house to play music". I said "whoaa, what mans house, I'm coming with you". So we all went to my daughter's friend's neighbor's house, and the bloke in there was a really dodgy looking 55+old man. The house was really messy, it stunk of weed, and the man was obviously an alcoholic. He had loads and loads of musical instruments. I said the children could play the instruments for a few minutes, and I started talking the man, he said "I love children, they come over all the time to play the instruments and I give them sweets or we play football".......I was like, ok then! We left, and I went to my daughter's friend's mum, and said "do you think it's ok to let you child visit him?" and my friend (the mum) said "oh yeah, he's fine all the children go over to his house because he give them sweets and lets them play his instruments, he's just eccentric" (I was surprised because this is a very smart women who is a teacher. Am I just being cynical? Or do you think that's weird? Is it ok to have older friiends for a child? What do you think of the whole thing?