As you could see in the drawing from the previous post, it's the ground the one that moves (actually, pretty fast). The building is the one who wants to stay calm and still, but the soil insists so much that, at the end, the house accepts to follow its motion. That's why you see its upper part arriving always "late". The earth moves left, while the building wishes to stay right, and it's not until later that the building moves (deforms, to be precise) to the left.
But then, when the building gets to the left, the ground is already returning to the right, something that the building will (later again, of course) do as well.
So the building moves. Later, but it moves. If you were the man standing under the tree you would see it completely different:
You would say that it was only the house the one that moved, and for you, its movement would be quite similar to a cantilevered beam. But, remember this?
Because you step on the earth, you see the building moving, though it intends, in fact, not to move at all. If there was another house close to it, you would even see how they pound on each other.
But remember: I feel the earth move under my feet!