Kryptonite

Kryptonite was a fictional crystaline substance that Superman was weak to. Colloquially, it was a synonym for a person's weakness.

Mike Yates asked the Fourth Doctor if the meteor was weakening him like Superman and Kryptonite. (AUDIO: Starfall)

Gwen Cooper knew H&M was like Kryptonite to Rhys Williams. (PROSE: The House That Jack Built)