Every aspect of your house interacts with every other part. You close a door: you change the pressure in another room. Turn on an exhaust fan: you pull conditioned tempered air out of the house and it must be made up somewhere else… and it will be.
The question is, from where? Air is lazy. It will take the least restrictive path inside – through the wet, moldy crawlspace below your basement, through the dusty attic, through the vents of combustion appliances themselves. I don’t know about you, but that is not the air I want my children to breathe.