I did not see the Prophet (may Allah's peace and blessings be upon him) wiping with his hand anything of the House except the two Yemeni Corners.

The Prophet (may Allah's peace and blessings be upon him) did not use to touch any of the four corners of the Ka‘bah except the Black Stone Corner and the Yamani Corner.
The House has four corners. The eastern corner has two merits:
1. It was built on the foundations of Ibrahīm.
2. It accommodates the Black Stone.
The Yamani Corner has one merit: It was built on the foundations of Ibrahīm. The Shami and Iraqi Corners do not have any of these merits. The Black Stone was taken out of the Ka‘bah from their side.
It is recommended to touch and kiss the Black Stone, and touch the Yamani corner without kissing it. However, it is not permissible to touch or kiss the other two corners.
The laws of Islam are predicated on compliance not innovation, and there is a divine wisdom behind everything in the Shariah of Allah.