I think it's unsurprising when founders of unicorns eschew A/B testing in favor of vision.
Admitting part of their success was luck is tantamount to admitting "I went from nothing to a billionaire because the dice came up right."
I don't know many people who would be comfortable truly accepting that. And even fewer who could do so AND be the type of driven person necessary. All while everyone around you is looking to you for answers.
Purely anecdotally, the one place I worked at that embraced A/B testing went to such an extreme that it stifled the vision. It became so risk averse that nothing was really happening, yet at the same time the business was not in the place to sustain growth or even profit that way.
This is because the introduction of such a data driven mentality meant the only people empowered to suggest decisions or experiments or approve the work were all in marketing. Because they were basically deciding what workload they wanted and of course you won’t put more on your plate if you don’t have to.
This isn’t to say it’s about data vs. vision. The original success is usually created in large part due to a healthy collaboration between the visionaries, the engineers, and whoever else is in the mix. Once you take one part of the team out of the equation and tend too heavily towards a particular bias then you might well have lost the secret to the initial success.
Your experience is similar to mine. I'm a marketing consultant so I get to work with a lot of different teams. People that either over-rely on Founder vision AND people that overly rely on data tend to not do well. It's why I've stopped saying data-driven and now say data informed. The best teams have a solid vision, but let the data guide the specific implementation of said vision. You need a healthy mix of both.
I really like "data-informed" as an alternative to "data-driven". I'm going to copy that! It is important to honestly acknowledge the limitations of the data.
A/B usual represents an incremental comparison from one situation to another, and it helps 'walk down the feature path' ... but when you need to change paths, A/B testing is not going to help, this is where a bit of great PM talent is going to work.
Moreover, the more subtle the change, or more the basis' are not very comparable (two different apps), the less A/B has value.
For example: Snap was considered 'too complicated' for many users, they needed to make it simpler. But by doing so, they may alienate a bunch of current users, with the objective of making the app more amenable to the masses. This is a serious strategic issue. They might have gone about it somewhat differently but it's a difficult thing to do in any case, and they had to make changes.
The existential issue for Snap's problems is not this app change, it's the fact that all of Facebook's products are trying to eat them. Instagram, WhatsApp, Messenger etc ... these are the real threats.
Admitting part of their success was luck is tantamount to admitting "I went from nothing to a billionaire because the dice came up right."
I don't know many people who would be comfortable truly accepting that. And even fewer who could do so AND be the type of driven person necessary. All while everyone around you is looking to you for answers.