Consider this statement by Paul:
I will very gladly spend and be spent for you; though the more abundantly I love you, the less I be loved. (2 Cor. 12:15)
This is what it means to serve as a pastor. It is not easy and, to tell the truth, it never really gets easier. You will know pain and turmoil and often will get more respect outisde the church than you do in. But this is often what it is.
If you can’t do this, don’t become a pastor.
If you have reached the point where you think you can’t do this anymore, stop being a pastor.
If you are willing to do this, with the Lord’s help of course, you will succeed in ministry even if you never pastor more than a small group of people.