Theory:Doctor Who television discontinuity and plot holes/Heaven Sent

The obvious one. If all the rooms in the castle reset - how is the Doctor able to slowly wear down the diamond wall over multiple incarnations?