By the One in Whose Hand my soul is! Either you command good and forbid evil, or Allah will soon send upon you a punishment from Him, then you will pray to Him but He will not respond to you

The Prophet's saying "By the One in Whose Hand my soul is" is an oath in which the Prophet swears by Allah. The souls are in Allah's Hand, Exalted and Glorified. He guides or misguides them, and He causes them to live or die, if He wills. Allah says: {And [by] the soul and He Who proportioned it. And inspired it [with discernment of] its wickedness and its righteousness.} [Sūrat Ash-Shams: 7-8]
So the souls are in Allah's hand alone. That is why the Prophet swore by it. He would also swear by his own soul: "By the One in Whose hand the soul of Muhammad is." That is because the soul of Muhammad (may Allah's peace and blessings be upon him) is the best of souls. Therefore, he would swear by it.
The Prophet then mentioned the thing on which he took the oath: to command good and forbid evil, which is a religious duty that is to be fulfilled. Otherwise, Allah will send a common punishment and not answer our prayers.