Recursion was the concept that ideas and concepts, as Nyssa put it, "fold back on themselves". Trying to come to grips with the concept, Tegan surmised that "if we had an index file we could look it up in the index file under 'index file'".
The Doctor later realised that Castrovalva itself was — as a mathematical construct of the Master — inherently recursive. The Master intended to trap the Doctor there as the recursion folded in on itself and the reality collapsed. (TV: Castrovalva)