HOW TO SOLVE A INVITE LIST?

My parents want to invite lots of people to our wedding cousins, and frineds, and the whole thing as do his parents but me and my future hubby only want like 10 or 20 people from each side, i am not being mean but to many people makes me nervous and theres a lot of family hating on his side as well as mine soo how can i tell them we only want around 40 people tops, without being mean and my in laws hating me?