How do I get rid of the smell from my fireplace when it is not in use ?

In warm weather the smell comes in my house when the fireplace is not being used . Closing the damper helps but not completely.