The one big reason why CEOs exist is trust. Trust from the shareholders that someone at the company is trying to achieve gains for them. Trust from vendors/customers that someone at the company is trying to make a good product. Trust from the employees that someone is trying to bring in the money to the company (even if it doesn't come to them eventually).
And that trust can only be a person who is innately human, because the AI will make decisions which are holistically good and not specifically directed towards the above goals. And if some of the above goals are in conflict, then the CEO will make decisions which benefit the more powerful group because of an innately uncontrollable reward function, which is not true of AI by design.
This sounds a lot like the specious argument that only humans can create "art", despite copious evidence to the contrary.
You know what builds trust? A history of positive results. If AIs perform well in a certain task, then people will trust them to complete it.
> Trust from vendors/customers that someone at the company is trying to make a good product.
I can assure you that I, as a consumer, have absolutely no truth in any CEO that they are trying to making a good product. Their job is to make money, and making a good product is merely a potential side-effect.
And that trust can only be a person who is innately human, because the AI will make decisions which are holistically good and not specifically directed towards the above goals. And if some of the above goals are in conflict, then the CEO will make decisions which benefit the more powerful group because of an innately uncontrollable reward function, which is not true of AI by design.