It is because we are social, communial creatures. Others’ actions help define our culture and our culture is the social air we breathe. They don’t want the culture to value what you enjoy to the degree they perceive you enjoying it.