A person or thing which is unpleasant, dishonorable, or unwanted tends to appear (or reappear), especially at inopportune times. Things that are done improperly eventually cause problems.
The ‘clipping’ of coins was rife in the Middle Ages, long before standardisation of the coinage was reliably enforced. This example from the reign of Edward I shows the degree of ‘badness’ that pennies then endured.
The term ‘bad penny’ was established enough in English by the late 14th century for it to have been used in William Langland’s famous prose poem The vision of William concerning Piers Plowman, 1370-90: