I originally had the same issue as you, and I know exactly what is happening. This is a little hard to explain in words, so let me know if it makes sense:
1) Your TV, obviously, has some inherent display lag.
2) Therefore, the animation that plays after you press the button will also have some display lag.
3) Therefore, it will look as though you pressed the button too late, even though it's actually the animation that is playing too late.
4) Therefore, you will doubt your results and re-run the test, trying to get it right. After multiple tries, you will become adept at compensating for your TV's input lag, hitting the button too early and ending up with a result of 0.
Try the calibration again, but this time, just hit the button when YOU think the time is right. Don't pay attention to whether it looks early or late on the TV, because if the TV was correct, there wouldn't be anything to calibrate.
Edit: Oh, and once you do this, you'll want to get audio out from your TV. You're relying on the TV to keep audio and video in sync. Your current method will work fine for audio, but then the video will be off sync.