Howling:Rewinding time stream...

Sorry if this has been brought up elsewhere. In the finale the impression seems to be that The Doctor is rewinding along his own time stream. However, visiting a young Amy Pond sees him in a time and place that wasn't part of his time stream. He didn't return that night; he showed up years later by mistake. Taking that into consideration, it seems The Doctor is rewinding along Amy's timestream (she's the common denominator), not his own.

Is this just an oversight by the writers or is there an explanation I just didn't catch?