[At least that silence makes the Doctor more mellow, because he knows the magnitude of what he's about to say.] Mr. White wanted you back. He wanted you as a part of his unsavoury affairs. So he'd come to see you unannounced.

When Jane was dying, he was there standing over the both of you. He didn't offer any assistance. Didn't roll her over so that she could breathe- it's all she would have required.