Article by Ray Williams, appearing in Psychology Today.

The job of a CEO has never been more challenging and rewarding. However, the job can be a lonely one despite the generous compensation, perks and attention. Boards, and CEOs are increasingly turning to engaging professional executive coaches to assist CEOs in their performance and growth and reduce attrition.

Why should CEOs have coaches now? Previous generations managed without them. Today’s president or CEO faces more pressures than ever. Business leaders are dealing with rapidly changing markets, technologies and workforces, increased financial and legal scrutiny . . . and more. Top executives who feel that they can handle it all by themselves are more likely to burn out, make poor decisions or make no decisions – potentially resulting in significant loss of opportunities, human resources and financial resources. The job of CEO is unique from several perspectives: No one else needs to hear the truth more, and gets it less from employees; no one else is the focus of criticism when things go wrong; no one else is the final decision maker on difficult and often lose-lose decisions; and finally, no one else enjoys the almost hero-celebrity status and rewards.

The success rate and longevity of top executives is vast different than a generation ago. In the past two decades, 30% of Fortune 500 CEOs have lasted less than 3 years. Top executive failure rates as high as 75% and rarely less than 30%. Chief executives now are lasting 7.6 years on a global average down from 9.5 years in 1995. According to the Harvard Business Review, 2 out of 5 new CEOs fail in their first 18 months on the job. It appears that the major reason for the failure has nothing to do with competence, or knowledge, or experience, but rather with hubris and ego and a leadership style out of touch with modern times. Research shows when someone assumes a new or different leadership role they have a 40% change of demonstrating disappointing performance. Furthermore, 82% of newly appointed leaders derail because they fail to build partnerships with subordinates and peers.

Sydney Finkelstein, author of Why Smart Executives Fail, researched several spectacular failures during a six-year period. He concluded that these CEOs had similar deadly habits of which most were related to unchecked egos. David Dotlich and Peter C. Cairo, in their book, Why CEOs Fail: The 11 Behaviors That Can Derail Your Climb To The Top And How To Manage Them, present 11 cogent reasons why CEOs fail, most of which have to do with hubris, ego and a lack of emotional intelligence. Call it overconfidence or ego, but powerful and successful leaders often distrust or feel they don’t need advice from anyone.

A study by Kelly See, Elizabeth Wolfe Morrison, and Naomi Rothman, published in Organizational Behavior and Human Decision, concluded one characteristic of powerful and successful leaders is high levels of self-confidence. Unfortunately, the researchers say, the higher the self-confidence, the less likely these leaders are open to advice and feedback. They also make the point that powerful leaders seldom get useful feedback in their organizations. Subordinates are loath to give bad news or critical feedback, and many boards are not diligent in seeing feedback for performance improvement, particularly relationships, as important as other things, such as financial results. See and her colleagues also contend that today’s leaders are under enormous stresses. These stresses often produce anxiety, fear and physical illness, which strong leaders are hesitant to divulge over concern about judgments that may be made about their capacities or longevity.

Why is this leadership crisis happening? One reason may be the gaps between how leaders see themselves and how others see them. Call it self-awareness. These blind spots can be career limiting. The wider the gap, the more resistance there is to change. It also makes it difficult to create a positive organizational culture where openness and honesty are not encouraged.

Good leaders make people around them successful. They are passionate and committed, authentic, courageous, honest and reliable. But in today’s high-pressure environment, leaders need a confidante, a mentor, or someone they can trust to tell the truth about their behavior. They rarely get that from employees and infrequently from board members.

Paul Michelman, writing in the Harvard Business Review Working Knowledge, cites the fact that most major companies now make coaching a core part of their executive development programs. The belief is that one-on-one personal interaction with an objective third party can provide a focus that other forms of organizational support cannot. A 2004 study by Right Management Consultants found 86% of companies used coaches in their leadership development program.

Marshall Goldsmith, a high profile coach of leaders in Fortune 500 companies and author of The Leader of the Future, argues leaders need coaches when “they feel that a change in behavior—either for themselves or their team members—can make a significant difference in the long-term success of the organization.”

Eric Schmidt, Chairman and CEO of Google, who said that his best advice to new CEOs was “have a coach.” Schmidt goes on to say “once I realized I could trust him [the coach] and that he could help me with perspective, I decided this was a great idea…” Mike Myatt says in his article, The Benefits of a Top CEO Coach, “Executives who rise to the C-suite do so largely based upon their ability to consistently make sound decisions. However while it may take years of solid decision making to reach the boardroom it often times only takes one bad decision to fall from the ivory tower. The reality is that in today’s competitive business world an executive is only as good as his/her last decision, or their ability to stay ahead of contemporaries and competitors.”

Douglas McKenna, writing in Forbes magazine, argues that the top athletes in the world, and even Barack Obama, have coaches. In his study of executive coaching, McKenna, who is CEO and Executive Director for the Center for Organizational Leadership at The Oceanside Institute, argues that executive coaches should be reserved for everyone at C-level, heads of major business units or functions, technical or functional wizards and high-potential young leaders.

Despite its popularity, many CEOs and senior executives are reluctant to report that they have a coach, says Jonathan Schwartz, one-time President and CEO of Sun Microsystems, who had an executive coach himself. Steve Bennett, former CEO of Intuit says, “At the end of the day, people who are high achievers—who want to continue to learn and grow and be effective—need coaching.”

John Kador, writing in CEO Magazine, argues that while board members can be helpful, most CEOs shy away from talking to the board about their deepest uncertainties. Other CEOs can lend a helping ear, but there are barriers to complete honesty and trust. Kador writes, “No one in the organization needs an honest, close and long term relationship with a trusted advisor more than a CEO.” Kador reports conversations with several high profile CEOs: “Great CEOs, like great athletes, benefit from coaches that bring a perspective that comes from years of knowing [you], the company and what [you] need to do as a CEO to successfully drive the company forward,” argues William R. Johnson, CEO of the H.J. Heinz Co., “every CEO can benefit from strong, assertive and honest coaching.” The cost of executive coaches, particularly a good one, is not cheap, but “compared to the decisions CEOs make, money is not the issue,” says Schwartz, “if you have a new perspective, if you feel better with your team, the board and the marketplace, then you have received real value.”

The much asked question about coaching is its ROI. The majority of studies including a major one by Joy McGovern and her colleagues at the research firm, Manchester, indicated that the executives who received coaching valued the service between $100,000 to $1 million ROI. Joyce Russell, the Dean of the Robert H. Smith School of Business at the University of Maryland contends that once considered a concern of an employee or executive was assigned a coach, now it is viewed as a privilege and a sign that the organization values the executive’s contributions and is willing to invest money in their future growth and development.

Robert Lee former President and CEO of the Center for Creative Leadership provided a research study for the Society for Industrial and Organizational Psychology regarding the use of executive coaches in organizations. He identified the most common areas of focus which included: dealing with paradox and ambiguity; shared power; personal visibility vs. private persona; interpersonal distance vs. personal closeness; narcissism and pride vs. humility; approachability vs. tough mindedness; emotional openness vs. rationality and logic; empowering and enabling vs. directive and forcefulness; extroversion vs. introversion; leading from the heart vs. leading from the head; ethics and morality vs. pragmatism and the ends justifies the means.