Don’t throw the baby out with the bathwater

return pic

Discard something valuable along with something not wanted.

origin pic

The earliest known reference to the phrase is in Thomas Murner’s Narrenbeschwörung (Appeal to Fools), which is dated 1512.